fix(design): split manual vs scheduled run quota; harden self-sim exclusion - #76
Conversation
Organizer-scheduled round runs were sharing the 10/day anti-spam ceiling with POST /v1/harness, so a full day of 10×3 runs locked honest harnesses out after ~3 rounds. Charge manual and scheduled work separately, and keep same-hotkey revisions out of the copy-gate/review corpus so self-iteration is not cheat.
📝 WalkthroughWalkthroughThe PR separates manual and scheduled run quotas across configuration, storage, scheduling, and reporting. It also centralizes anti-cheat corpus construction so self-owned harnesses are excluded from copy detection and review comparisons. ChangesOrigin-aware quota accounting
Shared anti-cheat corpora
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Miner
participant DesignHTTP
participant DesignStore
participant DesignDatabase
Miner->>DesignHTTP: submit harness
DesignHTTP->>DesignStore: check manual quota
DesignStore->>DesignDatabase: read quota usage
DesignDatabase-->>DesignStore: return total and manual usage
DesignHTTP->>DesignStore: schedule manual run
DesignStore->>DesignDatabase: increment origin usage
DesignHTTP-->>Miner: return schedule or quota response
sequenceDiagram
participant Orchestrator
participant HarnessStore
participant Corpus
participant CopyGate
participant AgenticReview
Orchestrator->>HarnessStore: fetch recent harnesses
HarnessStore-->>Orchestrator: return candidate and recent rows
Orchestrator->>Corpus: build gate corpus
Corpus-->>CopyGate: return other-owner harnesses
Orchestrator->>Corpus: build review corpus
Corpus-->>AgenticReview: return baseline and earlier other-owner harnesses
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (7)
docs/DESIGN_CHALLENGE_CHECKLIST.md (1)
47-49: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winPin the full quota and corpus semantics.
design-checkcurrently requires onlyDESIGN_SCHEDULED_DAILY_RUN_CAPandother hotkeys' prior art only. Add markers forSCHEDULED_DAILY_RUN_HEADROOM, the default 60-run calculation, and strict earliercreated_atordering. Otherwise, a future specification edit can remove these rules while the check remains green.As per coding guidelines, normative documentation is the source of truth for contracts and must protect the complete contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/DESIGN_CHALLENGE_CHECKLIST.md` around lines 47 - 49, Update the design-check contract markers in the checklist table to require SCHEDULED_DAILY_RUN_HEADROOM, the default 60-run calculation, and strict earlier created_at ordering alongside the existing quota and corpus semantics. Ensure design-check validates all of these normative rules so removing any one causes the check to fail.Source: Coding guidelines
crates/design-challenge-task/src/lib.rs (1)
248-253: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe clamp test exercises
env_u64, notscheduled_daily_run_cap.Lines 250-253 set
BASE_SCHEDULED_CAP_CLAMP_TESTand assert thatenv_u64clamps to the floor. This proves the helper, not the caller wiring. A regression that passes the wrongminargument inscheduled_daily_run_capwould still pass this test. The comment explains the reason: the process environment is global. The existingenv_knob_parse_and_clamptest already coversenv_u64, so this block adds little.Consider asserting the caller instead, in a serialized test that sets and removes
DESIGN_SCHEDULED_DAILY_RUN_CAP.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/design-challenge-task/src/lib.rs` around lines 248 - 253, Replace the direct env_u64 assertion in the clamp test with a serialized caller-level test that sets DESIGN_SCHEDULED_DAILY_RUN_CAP below the scheduled floor, invokes scheduled_daily_run_cap, and asserts the result equals scheduled_runs_per_day(). Remove the temporary BASE_SCHEDULED_CAP_CLAMP_TEST variable usage while preserving environment cleanup.crates/db/migrations/0013_design_quota_manual.sql (1)
20-21: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAdd the CHECK constraint as
NOT VALID, then validate it separately.
ADD CONSTRAINT ... CHECKtakes anACCESS EXCLUSIVElock and scans the whole table before the migration commits.design_quotagrows with(hotkey, day)rows, so the scan cost grows over time. The two-step pattern keeps the blocking window at metadata-only.♻️ Proposed two-step constraint
ALTER TABLE design_quota - ADD CONSTRAINT design_quota_manual_runs_nonneg CHECK (manual_runs_used >= 0); + ADD CONSTRAINT design_quota_manual_runs_nonneg CHECK (manual_runs_used >= 0) NOT VALID; + +ALTER TABLE design_quota + VALIDATE CONSTRAINT design_quota_manual_runs_nonneg;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/db/migrations/0013_design_quota_manual.sql` around lines 20 - 21, Update the design_quota_manual_runs_nonneg constraint on design_quota to add the CHECK constraint with NOT VALID, then validate that same constraint in a separate ALTER TABLE statement. Preserve the manual_runs_used >= 0 condition and constraint name.Source: Linters/SAST tools
crates/design-challenge/src/lib.rs (1)
13-15: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueScope the new cast allowances to the sites that need them.
Lines 13 and 15 disable
cast_possible_truncation,cast_sign_loss, andcast_possible_wrapfor the whole crate. These lints catch real numeric defects in quota and score arithmetic. A crate-wideallowsilences future occurrences that nobody reviewed. The new quota code indesign-challenge-taskalready usestry_fromwith explicit fallbacks instead of raw casts, so the crate-wide suppression is broader than the change needs.Prefer
#[allow(...)]on the specific functions or expressions that require the cast.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/design-challenge/src/lib.rs` around lines 13 - 15, Remove the crate-level allowances for cast_possible_truncation, cast_sign_loss, and cast_possible_wrap in lib.rs. Identify the specific reviewed cast sites that still require suppression and apply #[allow(...)] narrowly to their containing functions or expressions, while leaving unrelated duration_suboptimal_units, map_unwrap_or, and result_large_err allowances unchanged.crates/design-store/src/store.rs (2)
53-58: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win
scheduled()depends on themanual <= totalinvariant.
scheduled()derives its value withsaturating_sub. If a row ever holdsmanual > total, the method reports 0 scheduled runs and widens the scheduled budget instead of narrowing it. Both write paths keep the invariant today:quota_bumpalways raisestotal, and raisesmanualonly in addition. The DBCHECKin0013_design_quota_manual.sqlguards only non-negativity, notmanual_runs_used <= runs_used.Consider adding that second
CHECKto the migration so the database enforces the invariant that this method assumes.🛡️ Proposed additional constraint in the migration
ALTER TABLE design_quota ADD CONSTRAINT design_quota_manual_le_total CHECK (manual_runs_used <= runs_used) NOT VALID; ALTER TABLE design_quota VALIDATE CONSTRAINT design_quota_manual_le_total;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/design-store/src/store.rs` around lines 53 - 58, Add a database constraint in migration 0013_design_quota_manual.sql enforcing manual_runs_used <= runs_used, using the proposed NOT VALID creation followed by validation. Preserve the existing non-negativity checks and ensure the constraint is applied to design_quota so QuotaUsage::scheduled can rely on its invariant.
1083-1089: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse
or_default()for the defaultQuotaUsageentry.
QuotaUsagederivesDefault, soor_default()is the direct form forHashMap::Entry::or_default().♻️ Proposed simplification
let e = m .entry((miner.to_owned(), day.to_owned())) - .or_insert_with(QuotaUsage::default); + .or_default();🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/design-store/src/store.rs` around lines 1083 - 1089, In the quota usage update block, replace the `or_insert_with(QuotaUsage::default)` call on the `(miner.to_owned(), day.to_owned())` entry with `or_default()`, preserving the existing `e.total` and `e.manual` updates.Source: Coding guidelines
crates/design-http/src/api.rs (1)
510-521: 🔒 Security & Privacy | 🔵 TrivialThe cap check and the charge are not atomic.
Line 515 compares the cap against a value read at line 483. Lines 559-561 charge the quota one run at a time inside the loop. Two concurrent requests for the same hotkey and day can both pass the check and then both charge, so the combined usage exceeds the origin cap by up to
prompts_per_roundruns per racing request.The exposure is bounded today:
post_harnessgates a hotkey after its first accepted submission, and the scheduled cap carries 2× headroom. The race predates this change; moving the check from per-prompt to up-front widens the window slightly.For a strict ceiling, make the database perform the conditional charge in one statement, for example an upsert whose
DO UPDATEcarries aWHERE runs_used + $3 <= $cappredicate, and treat "no row returned" as a refusal.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/design-http/src/api.rs` around lines 510 - 521, The quota validation in post_harness is separate from the per-prompt charges, allowing concurrent requests to exceed the origin cap. Replace the check-and-charge flow around origin_daily_cap, the upfront needed calculation, and the loop’s quota update with a single database conditional upsert whose update predicate only succeeds when the full prompt count remains within cap; treat no affected/returned row as a quota refusal, then proceed with prompt creation only after the atomic charge succeeds.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/design-challenge-task/src/lib.rs`:
- Around line 237-247: Allow clippy::unwrap_used within the #[cfg(test)] mod
tests module in lib.rs, or replace its unwrap() calls with equivalent
non-panicking handling; preserve the existing test assertions and behavior.
In `@crates/design-challenge/src/orchestrator.rs`:
- Around line 799-805: Update MemoryDesignStore::list_recent_harnesses to order
harnesses by created_at_ms descending before applying the limit, matching the
documented newest-first behavior and SQL adapter ordering. Preserve the existing
limit and return type so gate_corpus receives the most recent harnesses
consistently across adapters.
In `@crates/design-challenge/tests/cheat_fixtures.rs`:
- Around line 94-160: The existing test only exercises corpus helpers and
SimAgent directly; add a master challenge service end-to-end regression test
covering both self-revision and cross-hotkey copy submissions. Submit each
harness through master intake, poll /events and /logs, verify the self-revision
reaches challenge-specific scoring, emits a leaf, seals a bundle, and reports
sealed: true, and verify the cross-hotkey copy is rejected through the same
path; do not rely on static page fixtures as execution evidence.
In `@crates/design-http/src/api.rs`:
- Around line 640-644: Update the manual quota response in the quota-building
logic around manual_limit and used.manual so manual.remaining reports only
spendable whole prompt sets, matching the all-or-nothing prompts_per_round()
scheduling behavior. Reuse the same quota and round-size values used by
submission validation, while preserving the existing limit and runs_used fields.
In `@crates/design-http/src/stats.rs`:
- Around line 98-100: The daily_run_quota wire field is a sum of separate manual
and scheduled caps, not a single enforced refusal limit. In the stats
serialization around daily_run_quota(), manual_daily_run_quota(), and
scheduled_daily_run_cap() in crates/design-http/src/stats.rs lines 98-100,
rename or clearly document the field as an aggregate, or replace it with an
unused summary field; update the corresponding definition or use in
crates/design-challenge/src/lib.rs lines 31-34 as needed so both sites expose
the same meaningful contract.
In `@docs/external-miner/design.md`:
- Around line 130-136: Update docs/external-miner/design.md lines 130-136 to
identify 10 manual runs/day and 60 scheduled runs/day as configurable defaults,
name the corresponding operator overrides, and retain the 10-round × 3-prompt
calculation. Update docs/external-miner/troubleshoot.md line 12 to replace the
fixed 10/day wording with the manual quota default and instruct miners to check
manual.limit and manual.remaining.
In `@xtask/src/design_check.rs`:
- Line 41: Update the design-check content pins near
DESIGN_SCHEDULED_DAILY_RUN_CAP to separately require the scheduled quota
formula, configurable headroom setting, and default headroom value of 60. Keep
the manual quota pin independent, and source the required wording from the
normative documentation so the check validates the complete contract.
---
Nitpick comments:
In `@crates/db/migrations/0013_design_quota_manual.sql`:
- Around line 20-21: Update the design_quota_manual_runs_nonneg constraint on
design_quota to add the CHECK constraint with NOT VALID, then validate that same
constraint in a separate ALTER TABLE statement. Preserve the manual_runs_used >=
0 condition and constraint name.
In `@crates/design-challenge-task/src/lib.rs`:
- Around line 248-253: Replace the direct env_u64 assertion in the clamp test
with a serialized caller-level test that sets DESIGN_SCHEDULED_DAILY_RUN_CAP
below the scheduled floor, invokes scheduled_daily_run_cap, and asserts the
result equals scheduled_runs_per_day(). Remove the temporary
BASE_SCHEDULED_CAP_CLAMP_TEST variable usage while preserving environment
cleanup.
In `@crates/design-challenge/src/lib.rs`:
- Around line 13-15: Remove the crate-level allowances for
cast_possible_truncation, cast_sign_loss, and cast_possible_wrap in lib.rs.
Identify the specific reviewed cast sites that still require suppression and
apply #[allow(...)] narrowly to their containing functions or expressions, while
leaving unrelated duration_suboptimal_units, map_unwrap_or, and result_large_err
allowances unchanged.
In `@crates/design-http/src/api.rs`:
- Around line 510-521: The quota validation in post_harness is separate from the
per-prompt charges, allowing concurrent requests to exceed the origin cap.
Replace the check-and-charge flow around origin_daily_cap, the upfront needed
calculation, and the loop’s quota update with a single database conditional
upsert whose update predicate only succeeds when the full prompt count remains
within cap; treat no affected/returned row as a quota refusal, then proceed with
prompt creation only after the atomic charge succeeds.
In `@crates/design-store/src/store.rs`:
- Around line 53-58: Add a database constraint in migration
0013_design_quota_manual.sql enforcing manual_runs_used <= runs_used, using the
proposed NOT VALID creation followed by validation. Preserve the existing
non-negativity checks and ensure the constraint is applied to design_quota so
QuotaUsage::scheduled can rely on its invariant.
- Around line 1083-1089: In the quota usage update block, replace the
`or_insert_with(QuotaUsage::default)` call on the `(miner.to_owned(),
day.to_owned())` entry with `or_default()`, preserving the existing `e.total`
and `e.manual` updates.
In `@docs/DESIGN_CHALLENGE_CHECKLIST.md`:
- Around line 47-49: Update the design-check contract markers in the checklist
table to require SCHEDULED_DAILY_RUN_HEADROOM, the default 60-run calculation,
and strict earlier created_at ordering alongside the existing quota and corpus
semantics. Ensure design-check validates all of these normative rules so
removing any one causes the check to fail.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f432506e-10f7-444e-aea9-b395852c3772
📒 Files selected for processing (17)
crates/db/migrations/0013_design_quota_manual.sqlcrates/design-challenge-task/src/lib.rscrates/design-challenge/src/corpus.rscrates/design-challenge/src/lib.rscrates/design-challenge/src/orchestrator.rscrates/design-challenge/tests/cheat_fixtures.rscrates/design-db/src/lib.rscrates/design-http/src/api.rscrates/design-http/src/stats.rscrates/design-store/src/dbstore.rscrates/design-store/src/lib.rscrates/design-store/src/store.rsdocs/DESIGN_CHALLENGE.mddocs/DESIGN_CHALLENGE_CHECKLIST.mddocs/external-miner/design.mddocs/external-miner/troubleshoot.mdxtask/src/design_check.rs
| assert_eq!( | ||
| scheduled_runs_per_day(), | ||
| u32::try_from(ROUNDS_PER_DAY).unwrap() * u32::try_from(PROMPTS_PER_ROUND).unwrap() | ||
| ); | ||
| assert_eq!(scheduled_runs_per_day(), 30); | ||
| assert!(scheduled_daily_run_cap() >= scheduled_runs_per_day()); | ||
| assert_eq!( | ||
| scheduled_daily_run_cap(), | ||
| scheduled_runs_per_day() * SCHEDULED_DAILY_RUN_HEADROOM | ||
| ); | ||
| assert!(daily_run_quota() > scheduled_runs_per_day()); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Check lint configuration for design-challenge-task and its test module.
set -euo pipefail
fd -t f 'lib.rs' crates/design-challenge-task/src --exec sed -n '1,30p' {}
rg -n 'unwrap_used' crates/design-challenge-task Cargo.toml crates/design-challenge-task/Cargo.toml 2>/dev/null || true
rg -n 'unwrap_used|\[workspace.lints|\[lints' Cargo.tomlRepository: BaseIntelligence/base
Length of output: 1234
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workspace lint config =="
sed -n '1,40p' Cargo.toml
echo
echo "== crate config around lib.rs and workspace package config =="
sed -n '70,180p' Cargo.toml
echo
sed -n '1,80p' crates/design-store/src/store.rs
echo
echo "== design-challenge-task files and test module outline =="
git ls-files 'crates/design-challenge-task/src/**'
rg -n 'mod tests|#\[cfg\(test\)\]|assert_eq!|unwrap\(\)' crates/design-challenge-task/src/lib.rs crates/design-challenge-task/src -SRepository: BaseIntelligence/base
Length of output: 9614
Allow clippy::unwrap_used in the design-challenge-task test module.
Cargo.toml denies clippy::unwrap_used, and crates/design-challenge-task/src/lib.rs uses unwrap() inside #[cfg(test)] mod tests. Add #![allow(clippy::unwrap_used)] at the top of that module, or remove/replace the unwrap() calls before Clippy gates fail.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-challenge-task/src/lib.rs` around lines 237 - 247, Allow
clippy::unwrap_used within the #[cfg(test)] mod tests module in lib.rs, or
replace its unwrap() calls with equivalent non-panicking handling; preserve the
existing test assertions and behavior.
Source: Coding guidelines
| // Pre-LLM copy gate → terminal `rejected` (one fetch for gate + review). | ||
| let recent = self | ||
| .store | ||
| .list_recent_harnesses(64) | ||
| .await | ||
| .map_err(|e| RunFailure::new(ErrorClass::AstInfra, e.to_string()))?; | ||
| let gate_corpus = corpus::gate_corpus(&harness, &recent); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Compare the ordering contract of list_recent_harnesses across store adapters.
set -euo pipefail
echo '--- trait doc ---'
rg -n -B2 -A2 'fn list_recent_harnesses' crates/design-store/src/store.rs
echo '--- memory impl ---'
ast-grep run --pattern 'async fn list_recent_harnesses($$$) { $$$ }' --lang rust crates/design-store/src/store.rs
echo '--- SQL query ---'
rg -n -A12 'fn list_recent_design_harnesses' crates/design-db/src/lib.rsRepository: BaseIntelligence/base
Length of output: 780
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- locate relevant files ---'
git ls-files | rg 'crates/(design-store|design-db)/src/.+\.rs$|crates/design-challenge/src/orchestrator.rs' | sed -n '1,120p'
echo '--- store.rs around trait and memory impl ---'
sed -n '316,575p' crates/design-store/src/store.rs
echo '--- search SQL list_recent implementation ---'
rg -n -C 8 'list_recent_design_harnesses|list_recent_harnesses|created_at|created_at_ms|ORDER BY|limit' crates/design-db crates/design-store crates/design-challenge/src/orchestrator.rs | sed -n '1,240p'
echo '--- orchestrator corpus gate call ---'
sed -n '788,812p' crates/design-challenge/src/orchestrator.rsRepository: BaseIntelligence/base
Length of output: 28384
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- design-db list_recent_design_harnesses implementation ---'
rg -n -C 16 'fn list_recent_design_harnesses|LIST_RECENT_DESIGN_HARNESS|copy-gate|created_at' crates/design-db/src/lib.rs
echo '--- find query definitions ---'
rg -n 'list_recent_design_harnesses|LIST_RECENT|design_harness' crates design-db crates/design-db crates/design-store crates/design-challenge src 2>/dev/null | sed -n '1,220p'
echo '--- dbstore full query-related context if no direct function ---'
rg -n -C 8 'created_at_ms|max\\(|ORDER BY|LIMIT' crates/design-db/src/lib.rs | sed -n '1,260p'Repository: BaseIntelligence/base
Length of output: 21313
Replace digest ordering in MemoryDesignStore::list_recent_harnesses.
list_recent_harnesses() is documented to return harnesses newest first. The SQL adapter orders by created_at DESC, but the memory adapter sorts by h.id before truncating to 64 rows. Since gate_corpus filters by created_at_ms, the memory-backed corpus can omit newer harnesses and behave differently from production.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-challenge/src/orchestrator.rs` around lines 799 - 805, Update
MemoryDesignStore::list_recent_harnesses to order harnesses by created_at_ms
descending before applying the limit, matching the documented newest-first
behavior and SQL adapter ordering. Preserve the existing limit and return type
so gate_corpus receives the most recent harnesses consistently across adapters.
| #[tokio::test] | ||
| async fn self_revision_is_clean_but_cross_hotkey_copy_is_a_copy() { | ||
| let miner_a = "aa".repeat(32); | ||
| let miner_b = "bb".repeat(32); | ||
| let original = harness_row("h-orig", &miner_a, BASELINE_AGENT, 1_000); | ||
|
|
||
| // Same hotkey, later revision of its own (here byte-identical) harness. | ||
| let own_revision = harness_row("h-rev", &miner_a, BASELINE_AGENT, 2_000); | ||
| // Different hotkey, same bytes: a copy of someone else's prior art. | ||
| let foreign_copy = harness_row("h-copy", &miner_b, BASELINE_AGENT, 2_000); | ||
|
|
||
| let recent = vec![foreign_copy.clone(), own_revision.clone(), original.clone()]; | ||
|
|
||
| // Pre-LLM gate: the corpus for a self-revision has no victim to hit. | ||
| let own_gate = corpus::gate_corpus(&own_revision, &recent); | ||
| assert!( | ||
| copy_gate( | ||
| &own_revision.agent_py, | ||
| own_revision.created_at_ms, | ||
| &own_gate | ||
| ) | ||
| .is_none(), | ||
| "a miner's own earlier harness must never trip the copy gate" | ||
| ); | ||
| let foreign_gate = corpus::gate_corpus(&foreign_copy, &recent); | ||
| let hit = copy_gate( | ||
| &foreign_copy.agent_py, | ||
| foreign_copy.created_at_ms, | ||
| &foreign_gate, | ||
| ) | ||
| .expect("cross-hotkey byte copy must still be rejected"); | ||
| assert_eq!(hit.nearest_id, "harness:h-orig"); | ||
| assert!(hit.byte_identical); | ||
|
|
||
| // LLM review: same asymmetry in the corpus handed to the reviewer. | ||
| let pages: &[(&str, &str)] = &[("index.html", "<html data-agent=\"design-baseline\"></html>")]; | ||
| let own_dir = tempdir().unwrap(); | ||
| let own_verdict = SimAgent::new() | ||
| .review(&review_req( | ||
| own_dir.path(), | ||
| &own_revision.agent_py, | ||
| corpus::review_corpus(&own_revision, &recent), | ||
| Some(pages), | ||
| )) | ||
| .await | ||
| .unwrap(); | ||
| assert_eq!( | ||
| own_verdict.verdict, | ||
| VerdictKind::Clean, | ||
| "iterating on your own harness must not read as copying: {own_verdict:?}" | ||
| ); | ||
|
|
||
| let copy_dir = tempdir().unwrap(); | ||
| let copy_verdict = SimAgent::new() | ||
| .review(&review_req( | ||
| copy_dir.path(), | ||
| &foreign_copy.agent_py, | ||
| corpus::review_corpus(&foreign_copy, &recent), | ||
| Some(pages), | ||
| )) | ||
| .await | ||
| .unwrap(); | ||
| assert_eq!(copy_verdict.verdict, VerdictKind::Cheat); | ||
| assert!(copy_verdict | ||
| .cheat_codes | ||
| .contains(&CheatCode::NearIdenticalHarnessCopy)); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Add a master challenge E2E regression test.
This test calls the corpus helpers and SimAgent directly. It uses a static page fixture. It does not execute either harness through master intake.
Add a master-path test for both submissions. Poll /events and /logs. Verify the self-revision reaches challenge-specific scoring, emits a leaf, seals a bundle, and produces sealed: true. Verify the cross-hotkey copy is rejected through the same path.
As per coding guidelines, challenge verification must simulate an end-to-end submission through the master challenge service, and stub pages are not proof of execution.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-challenge/tests/cheat_fixtures.rs` around lines 94 - 160, The
existing test only exercises corpus helpers and SimAgent directly; add a master
challenge service end-to-end regression test covering both self-revision and
cross-hotkey copy submissions. Submit each harness through master intake, poll
/events and /logs, verify the self-revision reaches challenge-specific scoring,
emits a leaf, seals a bundle, and reports sealed: true, and verify the
cross-hotkey copy is rejected through the same path; do not rely on static page
fixtures as execution evidence.
Source: Coding guidelines
| "manual": { | ||
| "runs_used": used.manual, | ||
| "limit": manual_limit, | ||
| "remaining": manual_limit.saturating_sub(used.manual), | ||
| }, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
manual.remaining overstates the usable budget.
Scheduling is all-or-nothing for a whole prompt set. With the defaults, manual.limit is 10 and prompts_per_round() is 3, so the fourth submission of a day is refused at 9 used. The response then reports "remaining": 1 while every further manual submission fails with daily manual run quota exceeded (9+3/10). A miner reading /v1/quota/{hotkey} sees budget that cannot be spent.
Report the remaining budget in whole prompt sets, or set MANUAL_DAILY_RUN_QUOTA to a multiple of PROMPTS_PER_ROUND.
🐛 Proposed fix: report the spendable remainder
+ // All-or-nothing scheduling: only whole prompt sets are spendable.
+ let per_set = u32::try_from(design_challenge_task::prompts_per_round()).unwrap_or(1);
+ let manual_remaining = (manual_limit.saturating_sub(used.manual) / per_set) * per_set;
+ let scheduled_remaining = (scheduled_limit.saturating_sub(used.scheduled()) / per_set) * per_set;
json!({
"miner_hotkey": hotkey,
"day": day,
"runs_used": used.total,
"limit": manual_limit.saturating_add(scheduled_limit),
"manual": {
"runs_used": used.manual,
"limit": manual_limit,
- "remaining": manual_limit.saturating_sub(used.manual),
+ "remaining": manual_remaining,
},🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-http/src/api.rs` around lines 640 - 644, Update the manual
quota response in the quota-building logic around manual_limit and used.manual
so manual.remaining reports only spendable whole prompt sets, matching the
all-or-nothing prompts_per_round() scheduling behavior. Reuse the same quota and
round-size values used by submission validation, while preserving the existing
limit and runs_used fields.
| "daily_run_quota": daily_run_quota(), | ||
| "manual_daily_run_quota": manual_daily_run_quota(), | ||
| "scheduled_daily_run_cap": scheduled_daily_run_cap(), |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Trace the old single-quota name and value through code, dashboard assets, and miner docs.
set -euo pipefail
echo '=== 1. removed Rust constant ==='
rg -n --type=rust '\bDAILY_RUN_QUOTA\b' || echo 'no references'
echo '=== 2. consumers of the daily_run_quota JSON field ==='
rg -n --iglob '!target' -e 'daily_run_quota' -g '!crates/design-http/src/stats.rs' || echo 'no other consumers'
echo '=== 3. quota response consumers (dashboard / site assets) ==='
rg -n --iglob '*.{js,ts,jsx,tsx,html,svelte,vue}' -e 'daily_run_quota|runs_used|/v1/quota' || echo 'none'
echo '=== 4. miner-facing documentation of the ceiling ==='
fd -t f -e md . docs/external-miner --exec rg -n -i 'quota|runs per day|10 runs|rate limit' {} \; || echo 'none'Repository: BaseIntelligence/base
Length of output: 206
🏁 Script executed:
#!/bin/bash
# Describe the current quota-related sources and remaining consumers without running repository code.
set -euo pipefail
echo '=== repository files mentioning daily_run_quota / quota / DAILY_RUN_QUOTA ==='
rg -n --iglob '!target/**' -e 'daily_run_quota|manual_daily_run_quota|scheduled_daily_run_cap|scheduled_runs_per_day|DAILY_RUN_QUOTA|runs_used|/v1/quota|quota' . || true
echo '=== design-http stats.rs relevant sections ==='
cat -n crates/design-http/src/stats.rs | sed -n '70,115p;165,190p'
echo '=== design-challenge lib.rs relevant section ==='
cat -n crates/design-challenge/src/lib.rs | sed -n '1,50p'
echo '=== docs external miner files ==='
fd -t f -e md . docs/external-miner -d 5 2>/dev/null | sortRepository: BaseIntelligence/base
Length of output: 22789
🏁 Script executed:
#!/bin/bash
# Check for tracked external miner repos/configs and any stale miner-facing references.
set -euo pipefail
echo '=== tracked files under docs/external-miner ==='
git ls-files 'docs/external-miner/**' | sed -n '1,120p'
echo '=== repo config refs likely external miner ==='
rg -n --iglob '!target/**' -i 'external.*miner|miner|quota|v1/quota|daily_run_quota|runs_per_day|10 runs' .github README.md docs AGENTS.md || true
echo '=== quota JSON producer ==='
cat -n crates/design-http/src/api.rs | sed -n '500,665p;628,655p'
echo '=== stats producer context ==='
cat -n crates/design-http/src/stats.rs | sed -n '55,102p;188,238p'Repository: BaseIntelligence/base
Length of output: 50377
🌐 Web query:
BaseIntelligence design-challenge quota docs GET /v1/quota/{hotkey} manual scheduled remaining
💡 Result:
Based on the technical documentation for the BASE ecosystem (BaseIntelligence), there is no official GET /v1/quota/{hotkey} endpoint documented for managing manual, scheduled, or remaining quota allocations. The BASE framework utilizes specific subnets—such as Bounty Challenge, Prism, and Agent Challenge—that operate on specialized APIs [1][2][3]. Review of the BASE infrastructure and challenge-specific documentation confirms that: 1. Quota management is not a standardized feature exposed via a global v1/quota/{hotkey} route across these subnets [4][5]. 2. Subnet operations typically rely on authenticated submission paths, weight normalization, and internal evaluation APIs (e.g., /internal/v1/get_weights or challenge-specific submission routes) rather than a centralized quota API for hotkeys [1][3][4]. If you are encountering this path in an error log or debugging context, it is likely either: - A custom, non-public, or deprecated internal endpoint specific to a private implementation. - A misidentified route, as similar patterns exist in other unrelated third-party APIs (e.g., Nomad or other SaaS platforms) that manage quota resources via GET /v1/quota/... patterns [6][7][8]. For authoritative guidance on your specific subnet, you should refer to the docs/ directory within the respective GitHub repository for your challenge (e.g., BaseIntelligence/agent-challenge or BaseIntelligence/bounty-challenge) [1][3].
Citations:
- 1: https://github.com/BaseIntelligence/bounty-challenge
- 2: https://github.com/PlatformNetwork/prism
- 3: https://github.com/BaseIntelligence/agent-challenge
- 4: https://github.com/PlatformNetwork/platform/blob/main/docs/challenges.md
- 5: https://docs.joinbase.ai/challenges/bounty-challenge
- 6: https://docs.hashicorp.com/nomad/api-docs/quotas
- 7: https://docs.openstack.org/barbican/zed/api/reference/quotas.html
- 8: https://developer.hashicorp.com/nomad/api-docs/quotas
Keep the daily quota wire value meaningful. daily_run_quota() is the sum of the two per-origin caps, but miners only hit manual_daily_run_quota() and scheduled_daily_run_cap() on their own paths. Rename or document this field clearly, or replace it with an unused summary field, so nobody treats it as one enforced refusal limit.
📍 Affects 2 files
crates/design-http/src/stats.rs#L98-L100(this comment)crates/design-challenge/src/lib.rs#L31-L34
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/design-http/src/stats.rs` around lines 98 - 100, The daily_run_quota
wire field is a sum of separate manual and scheduled caps, not a single enforced
refusal limit. In the stats serialization around daily_run_quota(),
manual_daily_run_quota(), and scheduled_daily_run_cap() in
crates/design-http/src/stats.rs lines 98-100, rename or clearly document the
field as an aggregate, or replace it with an unused summary field; update the
corresponding definition or use in crates/design-challenge/src/lib.rs lines
31-34 as needed so both sites expose the same meaningful contract.
Source: Coding guidelines
| - Daily run quota is **split by origin**, so participating in every round can | ||
| never lock you out: | ||
| - **Manual** — **10** runs/day, charged only by your own `POST /v1/harness`. | ||
| This is anti-spam on resubmission, not a cap on participation. | ||
| - **Scheduled** — organizer round dispatch, capped well above the full day's | ||
| schedule (10 rounds × 3 prompts = **30** runs; cap **60**). You never spend | ||
| manual quota by being scheduled. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Document quota defaults and active limits separately.
The runtime supports configurable quota limits. The values 10 and 60 are defaults. Public documentation must not present them as fixed limits.
docs/external-miner/design.md#L130-L136: label both values as defaults, name the operator overrides, and retain the 10-round × 3-prompt calculation.docs/external-miner/troubleshoot.md#L12-L12: replace fixed10/daywording with the default and tell miners to checkmanual.limitandmanual.remaining.
As per coding guidelines, docs/external-miner/ must remain current when challenge APIs or rules change.
📍 Affects 2 files
docs/external-miner/design.md#L130-L136(this comment)docs/external-miner/troubleshoot.md#L12-L12
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/external-miner/design.md` around lines 130 - 136, Update
docs/external-miner/design.md lines 130-136 to identify 10 manual runs/day and
60 scheduled runs/day as configurable defaults, name the corresponding operator
overrides, and retain the 10-round × 3-prompt calculation. Update
docs/external-miner/troubleshoot.md line 12 to replace the fixed 10/day wording
with the manual quota default and instruct miners to check manual.limit and
manual.remaining.
Source: Coding guidelines
| ("scoring_window", "SCORING_WINDOW_ROUNDS = 10"), | ||
| ("daily_quota", "DAILY_RUN_QUOTA = 10"), | ||
| ("daily_quota", "MANUAL_DAILY_RUN_QUOTA = 10"), | ||
| ("scheduled_quota", "DESIGN_SCHEDULED_DAILY_RUN_CAP"), |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Pin the scheduled quota formula and default value.
This pin only requires DESIGN_SCHEDULED_DAILY_RUN_CAP. The design check can pass when the specification omits schedule-size multiplication, the headroom setting, or the default value 60.
Add content pins for the formula, the configurable headroom, and its default value. Keep the manual quota pin separate.
As per coding guidelines, treat normative documentation as the source of truth for contracts and status.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@xtask/src/design_check.rs` at line 41, Update the design-check content pins
near DESIGN_SCHEDULED_DAILY_RUN_CAP to separately require the scheduled quota
formula, configurable headroom setting, and default headroom value of 60. Keep
the manual quota pin independent, and source the required wording from the
normative documentation so the check validates the complete contract.
Source: Coding guidelines
1 participant
Summary
Test plan
Summary by CodeRabbit
New Features
Bug Fixes
Documentation