feat(replay-verify): iterative in-sandbox fix loop with real-output feedback (opt-in) - #78
Merged
Conversation
…eedback --fix loop replaces the one-shot counterfactual fix with up to --fix-attempts attempts per case. Attempt 1 is byte-identical to the one-shot prompt; failed arms (nonzero exit or persistent signature) and failed model calls feed every prior command and its real executed stdout/stderr into the next prompt. Retries may answer with a short script (<=5 commands) executed as one /bin/sh unit. Every attempt runs in its own fresh sandbox with the same replayed prefix, so a flip always proves the corrected step against the recorded prefix state. The report adds fixFlipAttempt1 (one-shot-comparable), flipsByAttempt, the per-attempt trail on each case row, and armB-attempt<N>-result.json artifacts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The counterfactual patch-synthesis pursue:
--fix loop --fix-attempts k— attempt 1 is byte-identical to the shipped one-shot (test-asserted); on failure, attempt n+1 is prompted with the failed command's actual sandbox stdout/stderr, executed against a FRESH sandbox with the full replayed prefix (a flip always proves the corrected step against recorded state, never against a prior attempt's wreckage). Retries may emit ≤5-command scripts run as one arm. Default--fix generateunchanged; the loop is opt-in.Proof of mechanics (live, real sandbox + real glm-5.2)
Smoke on a case one-shot could not fix: three genuine feedback-chained attempts — a1 missing cargo env (exit 1, 27s) → a2 located cargo, drove a real build to an openssl-sys failure (exit 101, 398s) → a3 tried the missing package manager (exit 127). Exhausted honestly. 30 new tests (601/601 suite): flip@1 degeneration, flip@3, exhaustion, script cap, LLM-failure retry, per-attempt sandbox isolation.
Measurement status — honest
The pre-registered @1-vs-@3 comparison is not yet answerable: both full-batch attempts (run3, run4) hit z.ai seat degradation (28/33 and 29/33 call failures — 429 request-limits + long-reasoning aborts; run3 additionally overlapped another batch after an operator mutex error, ledgered). run4's surviving denominator (2/2 flips, both @1) is too small to claim anything. The canonical one-shot floor remains run2's 9/11. Cross-day replay stability is itself a finding: 16/22 (run2) vs 13/22 (run3, run4) replayable — 3 cases flip day-to-day; recorded for the replayability doc.
The comparison reruns on a healthy seat window with the committed command; the code is merge-ready independent of that readout.
Battery: typecheck clean, 601/601, build + package checks green, merge-tree clean.