fix(caller-drift): a fully remediated run is green, and the headline says so (backend#1965)

[LukasWodka]

Closes tracebloc/backend#1965

The bug

Remediation runs after the report is built and never touches findings, so a --create-prs run that opened a PR for every remediable copy still fell through to if findings: return 1. The watchdog headlines that Drift — indistinguishable from "drift exists and nobody has done anything".

The one-line fix would have made it worse

caller-drift.yml already refuses to let a clean exit code sit beside a non-zero findings count:

code == 0 and FINDINGS != 0
  -> "⚠️ Inconsistent result — the audit exited clean while reporting N finding(s).
      Treat fleet conformance as UNKNOWN and fix the audit."

So return 0 alone trades a false Drift for a false the-audit-is-broken — the worse of the two, because that headline tells the reader to stop trusting the audit. The exit code and the step outputs have to move together:

output	meaning
findings=	the un-remediated count — what the consistency guard should compare
findings_total=	the total, for the body
remediated=	how many ended the run with a PR open

And the verdict gains a branch, because a remediated run is green but is not "every entry matched" — the drift is real and still on the fleet until those PRs merge:

✅ Remediated — N drift finding(s), every one with a PR open. The fleet is NOT yet conformant, it is fixed-pending-merge: the next run goes red again if those PRs are closed unmerged.

Saying Conformant there would be this bug mirrored: a true exit code under a false headline.

standards-sync.py's rule is deliberately not copied

return 0 if args.create_prs else 1 is safe there because that tool has one homogeneous drift family. Here findings mixes callers, copies, protection, rulesets and quality, and only required copies are remediable — so a blanket rule would hide an unremediable protection or ruleset finding behind a green run. That is a fail-open in the guard written to refuse exactly that.

The predicate is measured instead: remediated is incremented beside the write that earns it, only when the PR succeeded, and green requires it to account for every finding there is.

Why this survived so long

The rule lived inline at the bottom of main(), reachable only by a full audit against the live org — so the selftest asserted nothing about it. It is now decide_exit(), a pure function, which is what makes it assertable.

11 new cases: the three the ticket names, plus the ones that keep the fix safe —

• partial remediation stays red and says how many are left
• one PR does not make nine findings green
• an unreadable read, or a failed remediation, still outranks a fully remediated one

Verification

selftest                173/173 (was 162)
make check              green
ruff                    clean
actionlint              clean on caller-drift.yml

The verdict logic was extracted from the workflow and executed over all six outcomes rather than read:

clean               -> ✅ Conformant
un-remediated       -> 🔴 Drift
FULLY remediated    -> ✅ Remediated  (was 🔴 Drift; naive fix -> ⚠️ Inconsistent)
partial             -> 🔴 Drift
remediation failed  -> ⚠️ Remediation failed
unreadable          -> ⚠️ Could not evaluate

Mutation-proved, anchors asserted:

remove the remediated branch            reddens the ticket's own case
relax `remediated >= findings`          reddens the nine-findings case
count entries whose PR FAILED           does NOT redden — see below

That third one is inert by construction, not uncovered: remediation_failures returns 2 before the remediated branch is reached, so a miscount there cannot produce a false green. Stated plainly rather than claimed as coverage.

Mutating the remediated branch out also printed -1 finding(s) remain, so the partial count is now clamped. Unreachable today — a nonsense count is how a real message stops being read.

Scope correction for the ticket

--create-prs only runs on a manual workflow_dispatch with the input set, never on the weekly cron, so this was never the "recurring org-wide audit" false red the ticket describes — one dispatch exists in the run history and it succeeded. Real, latent, and correctly last of the five.

Test plan

• python3 scripts/tests/caller-drift-selftest.py — 173/173
• make check — green
• ruff check scripts/caller-drift.py — clean
• actionlint .github/workflows/caller-drift.yml — clean
• Verdict logic extracted and executed across all six outcomes
• 2 reddening mutations with anchors asserted; 1 documented as inert by construction

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes audit exit semantics and org watchdog messaging for manual --create-prs runs; behavior is covered by new unit tests but affects how operators interpret green CI and the conformance issue body.

Overview
Fixes backend#1965: after --create-prs, remediation ran after the report and never reduced findings, so a run that opened a PR for every remediable copy still exited 1 and the watchdog showed Drift—the same headline as untouched drift.

decide_exit() in caller-drift.py centralizes the exit contract (0 = clean or fully remediated, 1 = un-remediated findings, 2 = unreadable / remediation failure). Green on remediation requires remediated >= findings (not a blanket create_prs green), so protection/ruleset/quality findings that no PR can fix stay red.

Step outputs now split counts: findings = un-remediated (for the workflow consistency guard), plus findings_total and remediated. remediated increments only when a copy remediation PR actually succeeds.

caller-drift.yml adds a Remediated verdict and findings breakdown when exit is 0 but PRs are open; the final job log distinguishes “fixed-pending-merge” from true conformant. Eleven selftest cases lock decide_exit() behavior (full/partial remediation, unreadable, failed writes).

^{Reviewed by Cursor Bugbot for commit 8c10915. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 463069e. Configure here.}

[shujaatTracebloc]

Approve (backend#1965). A fully-remediated --create-prs caller-drift run was falling through to if findings: return 1 and reporting a red "Drift" — indistinguishable from unfixed drift — because remediation happens after the report and never decrements findings. Fix extracts decide_exit as a pure function (so it is finally assertable, covered by caller-drift-selftest.py) with a clear contract: 2 = unreadable/remediation-failed (state UNKNOWN), 1 = findings remain un-remediated (partial says so in the numbers), 0 = nothing wrong OR every finding got a PR. New "✅ Remediated" verdict is green but explicitly NOT conformant (fixed-pending-merge, red again if those PRs close unmerged). The watchdog body and the run-log branch on the same REMEDIATED signal so the headline can never contradict the counts, and the existing FINDINGS-vs-total inconsistency guard correctly compares the un-remediated count. >= + clamp guard the nonsense-count edge (found by mutating the branch out). CI green, Bugbot clean.

[saqlainsyed007]

Approve (backend#1965). Correctness review — focused on the exit contract and its coupling to the watchdog.

What it does: extracts the exit rule into a pure decide_exit() and makes a fully-remediated --create-prs run exit 0 instead of falling through to if findings: return 1. Step outputs split into findings (un-remediated), findings_total, and remediated; the workflow gains a Remediated verdict and a matching final-log branch.

Verified the core invariant. Each required copy that is missing/drifted appends exactly one entry to findings and one to remediable, 1:1 (caller-drift.py:1994-1997, 2005-2013), and remediated += len(entries) counts only groups whose PR succeeded (2175-2181). So remediated <= len(findings) always, and remediated >= findings is true only when every finding is remediable and every one got a PR — non-remediable protection/ruleset/quality findings correctly keep the run red.

Coupling holds. Because decide_exit returns 0 only when the un-remediated count is 0, the output findings=max(len - remediated, 0) is 0 on every green run, so the workflow consistency guard (code == 0 && FINDINGS != 0) can never fire a false "Inconsistent result" against a remediated run. Verdict precedence (unreadable → remediation-failure → remediated → partial/un-remediated) is right in both decide_exit and the YAML.

Checks reproduced locally: caller-drift-selftest.py → 173/173. The 11 new cases pin the ticket case, partial-stays-red, over-count-stays-green, and unread/failed outranking remediated.

No correctness, convention, or coverage issues found. Bugbot's earlier "green log still claims full match" finding was addressed in 463069e.

[LukasWodka]

Thanks @shujaatTracebloc — approval received and the review itself stands.

Flagging a mechanical problem so it isn't mistaken for a stale check: this PR still reads REVIEW_REQUIRED / BLOCKED despite the approval, because shujaatTracebloc has read permission on tracebloc/.github and GitHub only counts approvals from users with write access toward a required approving review. So the approval cannot satisfy branch protection here, however many times it is given.

Added @saadqbal, who has write on this repo and is the CI/DevEx area reviewer, so the gate can actually be met. Nothing for Shujaat to redo.

Two things for @LukasWodka:

• On tracebloc/.github, saadqbal has write; shujaatTracebloc and saqlainsyed007 have read. Any PR here routed to the latter two dead-ends the same way — worth either granting write or pinning .github reviews to Asad.
• CODEOWNERS names @LukasWodka as sole owner and its own header notes he cannot approve his own PRs. require_code_owner_reviews is currently false on both develop and main, so that is not what is blocking this one — but it is the same single-gatekeeper shape, worth a look while you are here.