Skip to content

fix(orb): stop a conclusion-derived verdict from overwriting a recorded close - #8826

Merged
JSONbored merged 1 commit into
mainfrom
fix/gate-verdict-no-clobber
Jul 26, 2026
Merged

fix(orb): stop a conclusion-derived verdict from overwriting a recorded close#8826
JSONbored merged 1 commit into
mainfrom
fix/gate-verdict-no-clobber

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

What

Stops a gate verdict derived from the check conclusion from overwriting the verdict the bot actually acted on. Advances #8825 — this is the recording half; deciding how policy closes should be scored is the remaining part of that issue.

Why

Two callers write gate_decision, and both key the same deterministic row id (gate:<source>:<project>#<pr>@<sha>) with ON CONFLICT DO UPDATE — so the last writer wins:

  • processors.ts:3278 records the actual disposition the bot acted on (action: disposition.actionClass).
  • processors.ts:~10621 records a verdict derived from the gate conclusion alone, where success → merge (nativeGateActionFromConclusion).

On a PR the bot closed for a downstream reason (CI failure, contributor cap, policy), the conclusion-only writer runs last and clobbers the real close with merge. Concretely, #5861:

time event value
20:23:31 agent.action.close "CI is failing (validate, validate-tests (5))"
20:23:42 pr_outcome closed
20:23:44 gate_decision merge / success ← 13s AFTER the close

Fleet calibration reads the latest gate_decision as the gate's prediction, so every such row is scored as "the gate said merge and it ended up closed" — a false positive that never happened. The gate decided close, and the close was correct.

Measured on the live self-host across the merge-verdict-but-closed-outcome class (n=210): 84 were bot-closed and 59 carry a verdict timestamped after the close action it contradicts. This biases published accuracy downward — the opposite direction from the reversal under-counting in #8823, which is why no accuracy figure is currently trustworthy in either direction.

How

The DO UPDATE is guarded: it skips when the incoming write is conclusion-derived (input.action absent) and the stored decision is already close. A close is a terminal action that already happened; no later conclusion can un-close it, so the older row wins.

Deliberately narrow — the guard fires only for the exact contradiction observed:

  • an explicit action still replaces a stored close (a genuine later disposition change is still recorded);
  • non-close rows keep latest-finalize-wins unchanged;
  • no schema change, no migration, no behavior change to the gate itself.

Verification

  • TSC clean; 33 parity-wire tests and 398 across the downstream queue/outcomes suites, all green.
  • Three new regression tests: conclusion-derived never overwrites a close; an explicit action still does; non-close rows still update.
  • Changed-line coverage: 0 uncovered statements/branches.

Note on the historical rows

This stops new contradictions. The 59 already-recorded ones still misreport; quarantining or correcting them stays tracked on #8825 alongside the policy-close scoring decision.

…ed close

Both gate_decision writers key the same deterministic row id
(gate:<source>:<project>#<pr>@<sha>) with ON CONFLICT DO UPDATE, so the
last writer wins. One caller records the ACTUAL disposition the bot
acted on; the other derives the verdict from the gate-check conclusion
alone, where success maps to merge.

On a PR the bot closed for a downstream reason (CI failure, policy) the
conclusion-only writer runs last and clobbers the real close with a
merge. On #5861 the close landed at 20:23:31 and the
contradicting merge verdict was written at 20:23:44 -- 13 seconds after
the PR was already closed.

Fleet calibration reads the latest gate_decision as the gate's
prediction, so every such row is scored as a merge prediction that
ended closed: a false positive that never happened. Measured on the
live self-host, 59 rows carry a verdict timestamped after the close
action it contradicts, out of 210 in that class -- biasing published
accuracy DOWNWARD, opposite to the reversal under-counting in #8823.

Advances #8825 (the recording half; scoring policy closes as their own
class is the remaining part).

The DO UPDATE now skips when the incoming write is conclusion-derived
and the stored decision is already 'close'. A close is a terminal
action that already happened and no later conclusion can un-close it.
An explicit action still replaces it, and non-close rows keep
latest-finalize-wins.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 26, 2026
@JSONbored
JSONbored merged commit 6c1870c into main Jul 26, 2026
4 checks passed
@JSONbored
JSONbored deleted the fix/gate-verdict-no-clobber branch July 26, 2026 10:14
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.28%. Comparing base (4d6584d) to head (f85006d).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8826      +/-   ##
==========================================
- Coverage   93.83%   92.28%   -1.55%     
==========================================
  Files         800      800              
  Lines       79832    79833       +1     
  Branches    24197    24198       +1     
==========================================
- Hits        74909    73677    -1232     
- Misses       3558     5092    +1534     
+ Partials     1365     1064     -301     
Flag Coverage Δ
backend 92.96% <100.00%> (-2.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/review/parity-wire.ts 96.55% <100.00%> (+0.12%) ⬆️

... and 3 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant