feat(gdpval): resume multi-stage ELO from cache - #1933
Merged
Conversation
Make resume_from_cache work for multi-stage GDPval/stirrup ELO runs. The multistage driver bypassed the standard rollout-collection resume, so a rerun re-planned and re-executed every stage and lost stage results on a mid-run crash. - Persist a fingerprinted stage journal (plan + completion marker) so an interrupted stage replays the exact references/tasks it was using (deterministic even when multistage.seed is None). - Persist stage rollout rows incrementally as a stage-keyed instance of the core success/failure-sidecar contract (attempt-capped, terminal-aware); failures never contaminate the main jsonl or ELO. - Resume-aware run_multistage_stages: completed stages reuse cached judging results and re-fit ELO from rows; the interrupted stage re-dispatches only not-yet-done (stage, task, rollout) rows against the recorded references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
The multi-stage resume journal was only ever produced when a prior cache already existed: _prepare_resume returned None on a fresh run, and with resume=None the staged loop persisted nothing. So the first run never wrote <stem>_multistage_state.jsonl, making a later nel resume impossible (it re-ran every stage). A live nano-3.5 cluster run surfaced this — both stages completed but no journal was written. _prepare_resume now always returns a file-backed StageResume (writing callbacks), clearing stale files on fresh/stale/disabled paths and reusing prior state only on a valid fingerprint-matching resume. Fresh runs therefore persist the journal (plan at stage start, complete marker at stage end) and incremental rows, so resume has state to read. Regression test covers the fresh->resume cycle through _prepare_resume (the integration path the pragma had hidden). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make
resume_from_cachework for multi-stage GDPval/stirrup ELO runs. Previously the multistage driver bypassed the standard rollout-collection resume: a rerun re-planned and re-executed every stage, a crash lost in-progress results, and withmultistage.seedunset the adaptive task/reference sampling diverged on rerun.What changed
<stem>_multistage_state.jsonl): per-stageplannedrecord (chosen references, task ids, seed, prior ELO) written before dispatch + acompletemarker at stage end. A config fingerprint invalidates stale caches._failures.jsonl, kill-shaped → nowhere. Failures never contaminate the main jsonl or ELO.run_multistage_stages(pure, injectableStageResumeseam): completed stages reuse cached judging results and re-fit ELO from rows; an interrupted stage replays its recorded references (deterministic even whenseedisNone) and re-dispatches only not-yet-done(stage, task, rollout)rows. Per-task reuse within a partially-done stage is served by the stirrup agent's reference-subset-keyed/verifycache (rerun_incomplete)._prepare_resumenow always returns a writingStageResume, so a fresh run persists the journal + rows (previously nothing was written on the first run, making resume impossible).Test plan
resources_servers/gdpval/tests/— incl. fresh→resume cycle, interrupted-stage partial re-dispatch, plan replay withseed=None, failure routing (attempt cap / terminal), and fingerprint invalidation.[5, 5×num_models:2]):nel kill).nel resume→ run completed to SUCCESS.resuming multi-stage run from cache (fingerprint match)stage 1/2 reused from cache: eval ELO = 720.0 (5 cached rollout(s))— stage 1 not re-runstage 2/2 (replayed from journal): 5 task(s) vs 2 ref(s) ['gemma4_26b','gptoss_120b']— references replayed from the journal (withseed=null, only possible via persistence)🤖 Generated with Claude Code