Skip to content

v2.0.24

@that-guy-wade that-guy-wade tagged this 18 Sep 17:44
Prior behavior (post oro#320) only hard-failed when EVERY episode was a
harness failure. Today's Mistral outage exposed the flip side: OR's
shared upstream pool for mistralai/mistral-small-2603 (the simulator
model) was 429-rate-limited, and 60-80% of episodes came back as
environment_error. Runs succeeded with 0.03-0.10 partial scores and
counted against miners for infra we own.

Introduce _INFRA_FAILURE_THRESHOLD = 0.30. When
`environment_error + verifier_error >= 30%` of the roster, hard-fail
as `generated evaluation infrastructure failure` — the completed
sample isn't a representative slice of miner performance. The Backend
classifier already shields such failures from the auto-discard counter
and the work item re-opens so the run re-queues when infra recovers.

Below the threshold, harness episodes still score at zero reward
alongside agent_error — same as after oro#320. That path is exercised
by the existing `test_below_threshold_harness_failures_score_normally`
parametrized across [2, 3, 8] failed of 30 (max 26.7%).

Backend needs the mirror change in generated_scoring.py (separate PR)
since it re-computes the score for the mismatch check on complete_run.
Ship this PR first — validator will start hard-failing locally which
Backend already accepts as FAILED under the current 100% rule.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Assets 2
Loading