Skip to content

fix(livecodebench): a blank prediction is absent on disk, so read it with .get - #72

Closed
ethan-scitix wants to merge 1 commit into
mainfrom
fix/lcb-prediction-keyerror-on-resume
Closed

fix(livecodebench): a blank prediction is absent on disk, so read it with .get#72
ethan-scitix wants to merge 1 commit into
mainfrom
fix/lcb-prediction-keyerror-on-resume

Conversation

@ethan-scitix

Copy link
Copy Markdown
Collaborator

KeyError: 'prediction' when a lane is re-graded from disk

Re-grading a LiveCodeBench lane (sieval run --resume after clearing its terminal
records) raised exception::KeyError 'prediction' on 4 of 90 rollouts.

obj_to_dict drops None-valued keys, and PredictionRecord.prediction is
NotRequired[JSONValue | None] — so a rollout whose generation came back empty persists
as {"index", "extracted"} with no prediction key at all:

>>> json.loads(json.dumps(obj_to_dict(build_prediction_record([None]), False)))
{'rollouts': [{'index': 0, 'extracted': False}], ...}   # no 'prediction'

In-process the key is present and rollout["prediction"] works, which is why every fresh
run has been fine. It only fails after a reload. All four failures were truncations —
finish_reason: length, 0 chars of texts.

The task's own comment already said what it wanted — "an unextractable answer is None
here but '' on the wire"
— so .get("prediction") or "" is the intended read.

This is a family, not an incident

Across sieval/tasks/, 34 files subscript prediction directly and only 2 use
.get
. Essentially every generative lane fails the same way on a resume when some
rollout came back blank — silently, until someone re-grades. Resume is not a rare path:
it is what a harness bump makes people take.

Only the lane I could verify end to end is fixed here. The rest wants the same one-line
change plus a shared test, and reads better as its own pass than bundled into a bug
report — happy to do it either way.

Tests

tests/unit/tasks/test_livecodebench_prediction_reload.py, five cases. The premise is
asserted separately from the regression, so if serialization ever stops dropping the
key the first test says so rather than the suite quietly going vacuous.

Checked the regression test is not vacuous by reverting the fix — the subscript raises.
tests/unit/tasks: 380 passed, 1 skipped. ruff check / format --check clean.

How it surfaced

Verifying #66 end to end on a 1,000-question run. Two other things worth knowing came out
of the same exercise, both operational rather than code:

  • The code evaluator is a long-running service; git pull does not restart it, and
    the old request model silently drops an unknown timeout_per_case (pydantic ignores
    extras). A run graded with a whole-suite wall while looking like it honoured the new
    per-case rule.
  • Once the evaluator was restarted, per-case on identical generations gives
    48.89 → 46.67, 2 verdict flips (both case timeout: 6.0s), and timeouts carrying
    n_passed 7 missing → 0feat(livecodebench): replace the whole-suite timeout with upstream's per-case rule #66 working as designed.

🤖 Generated with Claude Code

…with .get

Re-grading a LiveCodeBench lane from disk (`sieval run --resume` after clearing its
terminal records) raised `exception::KeyError 'prediction'` on 4 of 90 rollouts.

`obj_to_dict` DROPS None-valued keys and `PredictionRecord.prediction` is
`NotRequired[JSONValue | None]`, so a rollout whose generation came back empty persists
as `{"index", "extracted"}` with no `prediction` key at all. In-process the key is there
and the subscript works, which is why every fresh run has been fine; it only fails after
a reload. All four were truncations -- `finish_reason: length`, 0 chars of `texts`.

The task's own comment already stated the intent ("an unextractable answer is None here
but '' on the wire"), so `.get("prediction") or ""` is what it meant to write.

Tests round-trip a record through `obj_to_dict` and assert the premise separately from
the regression, so if serialization ever stops dropping the key the first test says so
instead of the suite quietly going vacuous. Verified the regression test is not vacuous
by reverting the fix: the subscript raises. tests/unit/tasks: 380 passed, 1 skipped.

**This is a family, not an incident.** Across `sieval/tasks/`, 34 files subscript
`prediction` directly and only 2 use `.get`, so essentially every generative lane fails
the same way on a resume when some rollout came back blank -- silently, until someone
re-grades. Only the lane I could verify end to end is fixed here; the rest wants the same
one-line change plus a shared test, and is better done as its own pass than bundled with
a bug report.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ethan-scitix

Copy link
Copy Markdown
Collaborator Author

Closing as a duplicate of #70, which is strictly better and predates this by ~1.5 h.

#70 makes the identical one-line change to
livecodebench_code_generation_0shot_gen.py, and then does the two things this PR does
not: it fixes the other 38 sites, and it adds an enforcer
(check_preflight.py --check check_record_key_access) so the pattern cannot drift back.

Its diagnosis is also the right one, and it retires this PR's test too: the behaviour was
already pinned by test_records.py::TestSerializationRoundTrip:: test_none_prediction_is_absent_but_extracted_survives, which asserts exactly what the
test here asserts and has been passing the whole time. A record-level test constrains the
record, not its consumers — which is why 39 modules drifted with CI green, and why a
linter over the consumers is the fix rather than another round-trip test.

My mistake was not checking the open PRs before opening this one.

The one thing worth carrying over is field evidence, which I have posted on #70: this
fires in production, not just in review.

@ethan-scitix
ethan-scitix deleted the fix/lcb-prediction-keyerror-on-resume branch August 6, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant