feat(orb): deterministic decision replay — re-derive any gate decision from its recorded inputs - #9014
Merged
Merged
Conversation
…n from its recorded inputs Every evaluateGateCheck result now carries its own exact inputs (the advisory findings + the resolved policy), and the finalize site persists them beside the decision record in decision_replay_inputs (migration 0181) — a PRIVATE sibling table, because the public record's contract is digests-only for config and the findings carry contributor content. replayDecision re-executes the pure pipeline and compares conclusion, ordered blocker codes, and reasonCode (re-derived through the SAME deriveDecisionReasonCode the finalize now uses, so live and replay can never disagree about the mapping); the first divergent stage is reported and a divergence is a bug by definition. Replay cannot re-query the model or touch the network by construction — it is a pure function of two JSON values. Operator CLI: scripts/replay-decision.ts; CI smoke replays a fixed fixture corpus generated from the real evaluator.
Contributor
Logic backtestReplayed 0 historical case(s) for Backtest comparison:
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
❌ 3 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
JSONbored
added a commit
that referenced
this pull request
Jul 26, 2026
Two PRs merged within 18 minutes of each other and both grabbed migration number 0181 (0181_alert_dedup_claims.sql from #8985, 0181_decision_replay_inputs.sql from #9014), breaking db:migrations:check on main for every subsequent PR. Renumbers the later-merged file (decision_replay_inputs) to 0182, the next free number. No code or test references either file by name/number, so this is a pure rename.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #8838. Advances #8828.
Lands the deterministic replay core end to end; the two deliberately deferred pieces (re-query action-match mode, wall-clock capture for staleness rules) are split into their own precisely-scoped successor issue #9028 so this issue closes clean instead of lingering half-open.
Capture at the choke point:
evaluateGateCheckattachesreplay: {findings, policy}— its own exact inputs — to every evaluation, so no call site needs threading. The finalize persists them todecision_replay_inputs(migration 0181), deliberately a private sibling of the public record: the record's contract is config-digests-only and the findings carry contributor content; the publicrecord_digeststill commits the decision the replay explains. Synthetic evaluations (content-lane/bridge — their verdicts come from their own deterministic pipelines) carry no replay input and no-op inside the helper.Replay (
src/review/decision-replay.ts, pure): re-executesevaluateGateCheckand compares in pipeline order —conclusion→ orderedblocker_codes→reason_code— reporting only the FIRST divergent stage;actionis pinned (plan-state dependent).reasonCodeis re-derived throughderiveDecisionReasonCode, which the finalize site now ALSO uses — single source of truth, live and replay cannot drift. No-requery is structural:replayDecisionis a pure function of two JSON values.Operator CLI:
scripts/replay-decision.ts <bundle.json|->— exit 0 = replayed-same-verdict, 1 = divergence (structured stage diff; a bug by definition), 2 = unusable input. Extraction SQL in the header.CI smoke: four fixture bundles generated from the REAL evaluator replayed to match on every PR run, stage-1/2/3 divergence regressions, both persist arms, and an end-to-end queue test proving a live sweep's persisted record + replay row re-derive to
match.Changed-line coverage: all statements and branches.
decision_replay_inputsregistered in the schema-drift allowlist;ON CONFLICT(record_id) DO UPDATEis portable to self-host Postgres as-is.