fix(livecodebench): a blank prediction is absent on disk, so read it with .get - #72
fix(livecodebench): a blank prediction is absent on disk, so read it with .get#72ethan-scitix wants to merge 1 commit into
Conversation
…with .get
Re-grading a LiveCodeBench lane from disk (`sieval run --resume` after clearing its
terminal records) raised `exception::KeyError 'prediction'` on 4 of 90 rollouts.
`obj_to_dict` DROPS None-valued keys and `PredictionRecord.prediction` is
`NotRequired[JSONValue | None]`, so a rollout whose generation came back empty persists
as `{"index", "extracted"}` with no `prediction` key at all. In-process the key is there
and the subscript works, which is why every fresh run has been fine; it only fails after
a reload. All four were truncations -- `finish_reason: length`, 0 chars of `texts`.
The task's own comment already stated the intent ("an unextractable answer is None here
but '' on the wire"), so `.get("prediction") or ""` is what it meant to write.
Tests round-trip a record through `obj_to_dict` and assert the premise separately from
the regression, so if serialization ever stops dropping the key the first test says so
instead of the suite quietly going vacuous. Verified the regression test is not vacuous
by reverting the fix: the subscript raises. tests/unit/tasks: 380 passed, 1 skipped.
**This is a family, not an incident.** Across `sieval/tasks/`, 34 files subscript
`prediction` directly and only 2 use `.get`, so essentially every generative lane fails
the same way on a resume when some rollout came back blank -- silently, until someone
re-grades. Only the lane I could verify end to end is fixed here; the rest wants the same
one-line change plus a shared test, and is better done as its own pass than bundled with
a bug report.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Closing as a duplicate of #70, which is strictly better and predates this by ~1.5 h. #70 makes the identical one-line change to Its diagnosis is also the right one, and it retires this PR's test too: the behaviour was My mistake was not checking the open PRs before opening this one. The one thing worth carrying over is field evidence, which I have posted on #70: this |
KeyError: 'prediction'when a lane is re-graded from diskRe-grading a LiveCodeBench lane (
sieval run --resumeafter clearing its terminalrecords) raised
exception::KeyError 'prediction'on 4 of 90 rollouts.obj_to_dictdrops None-valued keys, andPredictionRecord.predictionisNotRequired[JSONValue | None]— so a rollout whose generation came back empty persistsas
{"index", "extracted"}with nopredictionkey at all:In-process the key is present and
rollout["prediction"]works, which is why every freshrun has been fine. It only fails after a reload. All four failures were truncations —
finish_reason: length, 0 chars oftexts.The task's own comment already said what it wanted — "an unextractable answer is None
here but
''on the wire" — so.get("prediction") or ""is the intended read.This is a family, not an incident
Across
sieval/tasks/, 34 files subscriptpredictiondirectly and only 2 use.get. Essentially every generative lane fails the same way on a resume when somerollout came back blank — silently, until someone re-grades. Resume is not a rare path:
it is what a harness bump makes people take.
Only the lane I could verify end to end is fixed here. The rest wants the same one-line
change plus a shared test, and reads better as its own pass than bundled into a bug
report — happy to do it either way.
Tests
tests/unit/tasks/test_livecodebench_prediction_reload.py, five cases. The premise isasserted separately from the regression, so if serialization ever stops dropping the
key the first test says so rather than the suite quietly going vacuous.
Checked the regression test is not vacuous by reverting the fix — the subscript raises.
tests/unit/tasks: 380 passed, 1 skipped.ruff check/format --checkclean.How it surfaced
Verifying #66 end to end on a 1,000-question run. Two other things worth knowing came out
of the same exercise, both operational rather than code:
git pulldoes not restart it, andthe old request model silently drops an unknown
timeout_per_case(pydantic ignoresextras). A run graded with a whole-suite wall while looking like it honoured the new
per-case rule.
48.89 → 46.67, 2 verdict flips (both
case timeout: 6.0s), and timeouts carryingn_passed7 missing → 0 — feat(livecodebench): replace the whole-suite timeout with upstream's per-case rule #66 working as designed.🤖 Generated with Claude Code