Skip to content

feat(orb): deterministic decision replay — re-derive any gate decision from its recorded inputs - #9014

Merged
JSONbored merged 2 commits into
mainfrom
feat/decision-replay-harness
Jul 26, 2026
Merged

feat(orb): deterministic decision replay — re-derive any gate decision from its recorded inputs#9014
JSONbored merged 2 commits into
mainfrom
feat/decision-replay-harness

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Closes #8838. Advances #8828.

Lands the deterministic replay core end to end; the two deliberately deferred pieces (re-query action-match mode, wall-clock capture for staleness rules) are split into their own precisely-scoped successor issue #9028 so this issue closes clean instead of lingering half-open.

Capture at the choke point: evaluateGateCheck attaches replay: {findings, policy} — its own exact inputs — to every evaluation, so no call site needs threading. The finalize persists them to decision_replay_inputs (migration 0181), deliberately a private sibling of the public record: the record's contract is config-digests-only and the findings carry contributor content; the public record_digest still commits the decision the replay explains. Synthetic evaluations (content-lane/bridge — their verdicts come from their own deterministic pipelines) carry no replay input and no-op inside the helper.

Replay (src/review/decision-replay.ts, pure): re-executes evaluateGateCheck and compares in pipeline order — conclusion → ordered blocker_codesreason_code — reporting only the FIRST divergent stage; action is pinned (plan-state dependent). reasonCode is re-derived through deriveDecisionReasonCode, which the finalize site now ALSO uses — single source of truth, live and replay cannot drift. No-requery is structural: replayDecision is a pure function of two JSON values.

Operator CLI: scripts/replay-decision.ts <bundle.json|-> — exit 0 = replayed-same-verdict, 1 = divergence (structured stage diff; a bug by definition), 2 = unusable input. Extraction SQL in the header.

CI smoke: four fixture bundles generated from the REAL evaluator replayed to match on every PR run, stage-1/2/3 divergence regressions, both persist arms, and an end-to-end queue test proving a live sweep's persisted record + replay row re-derive to match.

Changed-line coverage: all statements and branches. decision_replay_inputs registered in the schema-drift allowlist; ON CONFLICT(record_id) DO UPDATE is portable to self-host Postgres as-is.

…n from its recorded inputs

Every evaluateGateCheck result now carries its own exact inputs (the
advisory findings + the resolved policy), and the finalize site persists
them beside the decision record in decision_replay_inputs (migration
0181) — a PRIVATE sibling table, because the public record's contract is
digests-only for config and the findings carry contributor content.
replayDecision re-executes the pure pipeline and compares conclusion,
ordered blocker codes, and reasonCode (re-derived through the SAME
deriveDecisionReasonCode the finalize now uses, so live and replay can
never disagree about the mapping); the first divergent stage is reported
and a divergence is a bug by definition. Replay cannot re-query the
model or touch the network by construction — it is a pure function of
two JSON values. Operator CLI: scripts/replay-decision.ts; CI smoke
replays a fixed fixture corpus generated from the real evaluator.
@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (f5fe939) and head (92402af) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 26, 2026
@JSONbored
JSONbored merged commit 7e52ae4 into main Jul 26, 2026
5 of 6 checks passed
@JSONbored
JSONbored deleted the feat/decision-replay-harness branch July 26, 2026 16:09
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
22025 3 22022 21
View the top 3 failed test(s) by shortest run time
test/unit/check-engine-parity-script.test.ts > check-engine-parity script > runEngineParityMain returns 0 for the real monorepo workspace
Stack Traces | 0.0494s run time
AssertionError: expected 1 to be +0 // Object.is equality

- Expected
+ Received

- 0
+ 1

 ❯ test/unit/check-engine-parity-script.test.ts:798:48
test/unit/check-engine-parity-script.test.ts > check-engine-parity script > engine version skew > uses default version readers against the real monorepo workspace
Stack Traces | 0.0873s run time
AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "Gate-decision logic change in src/rules/advisory.ts requires either:
+   • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or
+   • a version bump in packages/loopover-engine/package.json (currently 3.15.0 vs base 3.15.0).",
+ ]

 ❯ test/unit/check-engine-parity-script.test.ts:785:31
test/unit/check-engine-parity-script.test.ts > check-engine-parity script > prints a clean summary and exits 0 for the real repo state when run as a subprocess
Stack Traces | 0.904s run time
Error: Command failed: .../loopover/node_modules/.bin/tsx scripts/check-engine-parity.ts
Engine-parity check found 1 issue(s):
Gate-decision logic change in src/rules/advisory.ts requires either:
  • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or
  • a version bump in packages/loopover-engine/package.json (currently 3.15.0 vs base 3.15.0).

 ❯ test/unit/check-engine-parity-script.test.ts:803:20

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { status: 1, signal: null, output: [ null, '', 'Engine-parity check found 1 issue(s):\nGate-decision logic change in src/rules/advisory.ts requires either:\n  • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or\n  • a version bump in packages/loopover-engine/package.json (currently 3.15.0 vs base 3.15.0).\n' ], pid: 7192, stdout: '', stderr: 'Engine-parity check found 1 issue(s):\nGate-decision logic change in src/rules/advisory.ts requires either:\n  • a matching edit to the other twin (.../src/advisory/gate-advisory.ts), or\n  • a version bump in packages/loopover-engine/package.json (currently 3.15.0 vs base 3.15.0).\n' }

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

JSONbored added a commit that referenced this pull request Jul 26, 2026
Two PRs merged within 18 minutes of each other and both grabbed migration
number 0181 (0181_alert_dedup_claims.sql from #8985, 0181_decision_replay_inputs.sql
from #9014), breaking db:migrations:check on main for every subsequent PR.

Renumbers the later-merged file (decision_replay_inputs) to 0182, the next free
number. No code or test references either file by name/number, so this is a
pure rename.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

trust: deterministic replay harness with the model call as the sole recorded nondeterministic input

1 participant