Repository navigation
Claims v2.0.0 introduces the weekly pipeline competition. Miners submit extraction packages before evaluation papers are revealed; the validator runs those packages, scores their outputs with the Claims Silver workflow, and selects a production pipeline from signed results. The release adds recurring rounds, parallel execution, auditable rule changes, and continuous weight updates.
Weekly competition
-
Committed packages and fresh evaluation papers. Each round freezes its submission rules, package commitments, model configuration and budgets before the paper sample is released. A deterministic round ID and a post-deadline random draw bind the evaluation to the published rules. (
51e94a2,13fd073,610e3e6) -
Continuous intake. The controller opens the next round while the current round is evaluated. It locks submissions, releases papers and advances scoring from durable round state, so a process restart can resume work without reopening intake or repeating completed stages. (
b1c7fc3,93d5a8a,01a7bf1) -
Staged evaluation. The first competition can use a single launch stage when there is no incumbent. Subsequent rounds can narrow the field through prescreen and screen before a final evaluation on fresh papers. Early stages can use Decision API routing for eligible judgments; final judging follows the frozen judge and tiebreak settings. (
610e3e6,82d998f,50802d2)
Scoring and selection
-
The established Silver workflow. Competition outputs pass through Claims comparison, eligibility adjudication, canonicalization and Silver scoring. The reference pipeline produces Bronze once per paper. Each challenger is scored against the baseline on the same paper and the same Silver record. Shared equivalent-claim credit is calculated within that pair, which can give the baseline a different paired score against different challengers. (
82d998f,c96b51d,07c1f19) -
A measured replacement rule. A challenger replaces the incumbent only when its average paper-by-paper gain exceeds 2.13 standard errors and it passes the applicable code and copy checks. The signed competition result determines the winner; consensus review runs separately and does not delay selection. (
69c6dde,cf3a366,c9ec705) -
Entry integrity without a round-wide hold. Team relationships and duplicate packages are checked before advancement. An entrant awaiting funding evidence is held individually, allowing eligible entrants to continue. Duplicate appeals are reviewed without pausing evaluation. (
bf0886a,38598bf,ec7a1c1)
Performance and reliability
-
Parallel, budgeted execution. Workers run multiple package–paper jobs within configured memory and concurrency limits. Bronze preparation and scoring can proceed as soon as papers are ready. Inference usage remains accounted for across retries, and live gateway counters enforce each paper's allowance. (
a570c9c,286277d,0d5c18f,1943eae) -
Traceable rule revisions. When a frozen model or evaluation rule needs correction, an audited amendment records the change and its effective stage. The controller resumes approved work while protecting completed results from silent reinterpretation. (
29236b7,0f75fd3) -
Reference response-cache protection. The DSPy ReAct runtime no longer caches raw model responses before validating them, preventing a malformed or truncated completion from being replayed across attempts. This change applies to reference packages built from the updated source; already frozen packages retain their original behavior. (
f0fc9f0)
Rewards
The signed weekly result supplies the competition weight vector. A separate heartbeat refreshes weights every six hours; until the first competition result is published, it carries forward the previous signed extraction vector. (25f94a5, 7d5b68c)