feat(campaign): GEPA round 2 infrastructure + cross-family certification — challenger rejected, first OpenHands/Terminus2 numbers - #523
Merged
Conversation
…gned feedback Pool grows to dev-32 + holdout-1 labeled positives (32 scenarios, both spent), split 20 train / 12 selection, seeded and stratified by gold-block width (widest contiguous run of labeled incorrect steps; wide >= 3). Feedback is micro-aligned two ways: the judge composite is multiplied by a gold-mass weight (labeled-issue count / pool max) so the mean GEPA climbs tracks the micro headline, and every evaluation returns per-case TP/FP/FN counts plus missed issue ids through the artifact evidence for reflection. The selection readout reports weighted, plain macro composite, plain macro f1, and pooled micro f1 side by side. Budget: 60 evaluations, proposer cap $5, cost ceiling $20. --plan mode writes the split plan with zero spend.
…hipping prompt stands; first OpenHands/Terminus2 numbers Pre-registered stock-vs-G2 certification on the freshly sealed OH-cert32 and T2-cert32 splits: the GEPA round-2 challenger (selection edge +0.064 weighted) was REJECTED — pooled-cert micro 0.1928 vs stock 0.2489, the third consecutive tuning-vs-fresh reversal. The stock arms are the analyst's first cross-family measurement: OpenHands 0.2896 / Terminus2 0.2162 micro vs 0.6288 on mini-SWE cascade gold — the ~2x family gap is the largest measured open lever. Both cert32 splits now spent for this configuration family.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
GEPA round 2 ran end-to-end and its winner lost the pre-registered certification — this PR ships the infrastructure, the honest negative, and the two most valuable numbers the campaign has produced: the shipping analyst's first cross-family measurements.
Commits
2e13fe8— campaign script round 2: 32-scenario pool (dev-32 + holdout-1 positives), width-stratified 20/12 train/selection split, gold-mass-weighted judge composite + per-case TP/FP/FN counts fed to GEPA reflection, contract frozen verbatim as round 1.3378a6d— provenance pin for the round-1 certified run.535b38b— the certification record:benchmarks/trace-analysis/codetracebench-crossfamily-cert-20260802/(pre-registration, canonical stock-arm results, rejected-arm sha pins, ledger).The experiment
60 evaluations, $7.05, glm-5.2 task + reflection. Winner changed (sha
1bb303e7…), selection edge +0.064 weighted / micro 0.400→0.509 (n=12, spent-split). Pre-registered certification on the freshly sealed OpenHands-cert32 + Terminus2-cert32 splits (seeded stratified cuts from the 716-row import, positionally calibrated before sealing):Verdict per rule 1: challenger REJECTED; the shipping prompt stands. Third consecutive tuning-vs-fresh reversal this campaign — the gold-mass-weighted selection metric is recorded as overfit-prone at n=12.
The numbers that matter beyond the reject
First-ever cross-family measurement of the shipping analyst: 0.29 OpenHands / 0.22 Terminus2 vs 0.63 mini-SWE cascade gold — it generalizes with a ~2× family gap, now the largest measured open lever, with 652 tuning-legal fresh rows available to attack it.
Verification
Battery on the branch tree: biome clean (657 files), typecheck clean, vitest 4656 passed / 3 skipped, digest checker green. Certification: 4 strictly-serial runs, $31.07 measured, all four result.json artifacts durable (
~/bench-cache/ctb-20260801/cert2/), rejected arms pinned by sha256 in the README. Evidence ledger updated (.evolve/experiments.jsonlrounds mp-r2 / mp-r2-cert).