Skip to content

Recover partial results when a lever generation call fails#292

Merged
neoneye merged 1 commit intomainfrom
fix/partial-result-recovery
Mar 16, 2026
Merged

Recover partial results when a lever generation call fails#292
neoneye merged 1 commit intomainfrom
fix/partial-result-recovery

Conversation

@neoneye
Copy link
Member

@neoneye neoneye commented Mar 16, 2026

Summary

  • When call 2 or 3 of the 3-call lever loop fails, keep levers from prior successful calls instead of discarding everything
  • Previously a single validator rejection (e.g., one lever with 2 options) wiped out 10+ valid levers from prior calls, causing total plan failures
  • If call 1 fails (no accumulated responses), the error is still raised
  • Companion fix to PR Add options count and review format validators to Lever #289 (option/review validators), which the iteration 13 assessment flagged as CONDITIONAL on this change

Test plan

  • CI passes
  • Run experiments across all 7 models x 5 plans
  • llama3.1 silo and gta_game should no longer produce zero output (they had valid levers from earlier calls)
  • haiku silo should recover levers from the 2 calls that succeeded before the editorial-commentary lever
  • Success rate should recover to >= 84% (current) and ideally approach 92% (pre-validator baseline)
  • No malformed levers in shipped artifacts (validators still active)

🤖 Generated with Claude Code

Previously, if call 2 or 3 of the 3-call lever loop failed (e.g., a
single lever with 2 options triggering the validator), all accumulated
levers from prior successful calls were discarded. This converted
single-lever validation errors into total plan failures.

Now, if at least one prior call succeeded, the loop breaks and returns
the levers collected so far instead of re-raising the error. If call 1
fails (no accumulated responses), the error is still raised since there
is nothing to recover.

This is the companion fix to PR #289 (option/review validators) which
the assessment flagged as CONDITIONAL on this change.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@neoneye neoneye merged commit 8395635 into main Mar 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant