Record blocked SOTA-v3 design and route catalog - #103
Conversation
|
Warning Review limit reached
Next review available in: 46 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughSOTA-v3 preregistration now records a blocked power design, while public model and pricing catalogs are frozen as metadata-only snapshots. Statistical tooling adds exact familywise simulation and optimized sign-flip calculations. Seed commitment tooling adds protected secret creation, ordered execution hashing, and validation gates. ChangesSOTA-v3 readiness controls
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant panel_power
participant ResultFiles
participant sign_flip_p_value
participant holm_adjust
panel_power->>ResultFiles: Load reference lift cells
panel_power->>panel_power: Simulate allocation and sensitivity trials
panel_power->>sign_flip_p_value: Calculate contrast p-values
panel_power->>holm_adjust: Apply Holm correction
holm_adjust-->>panel_power: Return familywise rejection metrics
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/test_sota_v3_route_catalog.py (1)
45-53: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAssert authorization flags are exactly
False.Truthiness passes for
0,None, or"", which would let a malformed registry look "locked". The sibling assertions in this file useis False.♻️ Proposed tightening
- assert not any( - registry[key] - for key in ( - "spend_authorized", - "route_preflight_authorized", - "panel_execution_authorized", - "publication_authorized", - ) - ) + for key in ( + "spend_authorized", + "route_preflight_authorized", + "panel_execution_authorized", + "publication_authorized", + ): + assert registry[key] is False, key🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_sota_v3_route_catalog.py` around lines 45 - 53, Update the authorization-flag assertion in the registry test to require each listed value to be exactly False, matching the sibling assertions, rather than merely falsy; preserve the existing keys and locked-state validation.tests/test_seed_panel_commitment.py (1)
98-105: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winOptional: cover the remaining
execution-hashrejection branches.The count-mismatch and committed-preset-overlap guards in
scripts/seed_panel_commitment.py(Lines 170-176) are the two gates that actually protect panel secrecy/identity, and neither is exercised. Two moreassert main([...]) == 2cases would close that.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_seed_panel_commitment.py` around lines 98 - 105, Extend test_execution_hash_rejects_missing_or_low_entropy_seed_env to cover the execution-hash count-mismatch and committed-preset-overlap rejection branches in commitment_mod.main. Add separate seed-panel environment/lane setups that trigger each guard and assert main([...]) returns 2, preserving the existing missing and low-entropy cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/sota_v3_pricing_snapshot.json`:
- Around line 22-97: The sota-v3 cost estimation flow in
scripts/estimate_publication_cost.py currently ignores model-specific
internal_reasoning and long_context_override pricing from the models in
sota_v3_pricing_snapshot.json. Update the estimator to apply internal_reasoning
rates for Gemini and threshold-based long_context_override prompt/completion
rates for OpenAI, Qwen, and xAI; alternatively, explicitly document that
expected_output_tokens_per_decision already includes those costs.
---
Nitpick comments:
In `@tests/test_seed_panel_commitment.py`:
- Around line 98-105: Extend
test_execution_hash_rejects_missing_or_low_entropy_seed_env to cover the
execution-hash count-mismatch and committed-preset-overlap rejection branches in
commitment_mod.main. Add separate seed-panel environment/lane setups that
trigger each guard and assert main([...]) returns 2, preserving the existing
missing and low-entropy cases.
In `@tests/test_sota_v3_route_catalog.py`:
- Around line 45-53: Update the authorization-flag assertion in the registry
test to require each listed value to be exactly False, matching the sibling
assertions, rather than merely falsy; preserve the existing keys and
locked-state validation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ee8a8f62-78d8-4595-86e3-e5656ea27867
📒 Files selected for processing (19)
config/sota_v3_lane.jsonconfig/sota_v3_models.jsonconfig/sota_v3_pricing_snapshot.jsonconfig/sota_v3_publication_protocol.jsondocs/PUBLISH_READINESS.mddocs/production_benchmark.mddocs/run_logs/sota-v3-preflight-2026-07-27.mddocs/run_logs/sota-v3-route-catalog-freeze-2026-07-28.mddocs/run_logs/sota-v3-statistical-design-audit-2026-07-28.mdgm_bench/publication.pyscripts/analyze_publication_panel.pyscripts/panel_power.pyscripts/seed_panel_commitment.pytests/test_panel_power.pytests/test_publication_analysis.pytests/test_publication_runner.pytests/test_seed_panel_commitment.pytests/test_sota_v3_preregistration.pytests/test_sota_v3_route_catalog.py
Outcome
Records the no-spend SOTA-v3 preregistration work without claiming execution readiness.
Validation
Remaining gates
Before any paid work: choose a viable statistical design or revise the preregistered claim; prove authenticated account route access and privacy/retention behavior; resolve the Luna reasoning-policy ambiguity with an explicitly authorized strict smoke; freeze the output token cap and cost ceiling; then separately authorize private seed generation, smoke spend, and panel spend.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Validation