feat(driver): error-grounded reflection — gepaDriver targets real failures (0.70.0)#146
Merged
Merged
Conversation
…r (0.70.0) Adversarial verification on TWO domains (legal + tax, two worker models): the gepaDriver's candidates REGRESSED the baseline (gate correctly held, nothing improved). Root cause: it reflected on per-scenario SCORES only — the judge's notes (the 'why it failed') were computed but DROPPED before the reflection, so it proposed generic rewrites a capable model already knows. Thread judge notes through generically: campaignBreakdown collects per-scenario notes (deduped) -> GenerationCandidate.scenarios[].notes -> gepaDriver buildEvidence -> TrialTrace.failureNote -> buildReflectionPrompt renders a 'Why it scored low' block. The optimizer now targets the real failure pattern. Anti-overfit: notes are GENERALIZABLE patterns by contract (not case ground truth — that's memorization), and the held-out gate is the structural backstop (overfit can't clear the paired-bootstrap CI on unseen cases). Generic — any agent benefits by emitting informative judge notes. 3 tests; suite 1645 green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The conjunct-2 fix. Adversarial verification on legal + tax (two worker models) showed the gepaDriver's candidates regressed — it reflected on per-scenario scores only; the judge's
notes(the 'why') were dropped before reflection, so it proposed generic rewrites that hurt a capable model.Threads judge
notesthrough generically:campaignBreakdown→scenarios[].notes→buildEvidence→TrialTrace.failureNote→ a 'Why it scored low' block in the reflection prompt.Anti-overfit (Drew's guardrail): notes are generalizable failure patterns by contract, never case ground-truth (that's memorization); the held-out gate is the structural backstop. Generic — any agent benefits. 3 tests; full suite 1645 green.