Skip to content

Reconcile the chained-event count when a run reports twice - #111

Merged
b-macker merged 2 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m
Aug 1, 2026
Merged

Reconcile the chained-event count when a run reports twice#111
b-macker merged 2 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m

Conversation

@b-macker

@b-macker b-macker commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Summary

A keyed live run showed L24-06 still failing after #109:

BREAK in run 1785542207246-30729: RunEnd declares 737 chained events but 785 observed

#109 was confirmed by that same run — every small run group came back diff 0, exactly where the end-of-run health warnings fire. But I called that writer the complete explanation, and it was not. Every large run in the earlier sample balanced exactly (628/628, 485/485), which is consistent with one writer at unit 2 — only because none of those runs tripped the quality gate. The second cause was invisible in that sample, not absent from the system.

Root cause

writeReports() is called from ~17 sites, and a clean execute() is not the last of them: main.cpp's contract-error, quality-gate and baseline-regression exits all call it again after the VM has already written and sealed the run.

The five file report formats are idempotent — they truncate and rewrite. Telemetry appends, and check_results_ is never cleared, so the second call re-emitted the entire result set after RunEnd had already declared the count. run_end_emitted_ then stopped the declaration from ever catching up.

The residual is exactly one second dump: GovernanceCheck 38 + RuleViolation 8 + GovernanceCheckSummary 1 + ScoringSnapshot 1 = 48.

The double-call was already known. The comment // writeReports() already called inside execute() on success sits three lines above two more call sites that do it anyway.

Changes

Fixed at the invariant, not the instance:

  • emitRunEnd() is no longer write-once. It re-emits whenever chained events followed the previous anchor. The verifier reads the last RunEnd per run_id, so the newest declaration reconciles the count — and any future writer that appends after an anchor is self-healing. This is the general form of both this defect and Stop the chain verifier from reporting tampering on its own output #109's.
  • The dump resumes from telemetry_results_dumped_ instead of restarting, so each result is written once.
  • telemetry_dedup_seen_ is now a member, so the dedup guarantee still holds across the multiple writeTelemetry() calls one run makes.
  • ScoringSnapshot re-emits only when the score moved, instead of filing an identical snapshot per call.

Beyond the verifier

The duplicate records mattered for more than chain accounting: every count derived from the telemetry file was inflated, including L25-03's taint total, which had been reading roughly double the true number.

Attributed by source location, the same run has 8 distinct sink sites and 7–8 violations per run segment — which also contradicts the "four sites" my own comment claimed. That comment is corrected to the real eight. The baseline stays at 18 as a loose ceiling, marked for re-measurement on a keyed run built after this fix; deliberately not halved by arithmetic here, since the fix changes what gets counted and only a run can say by how much.

Test Plan

  • Ran bash run-all-tests.sh with no new failures — 441 tests, 0 unexpected failures
  • Added/updated tests for new functionality — test_evidence_chain.sh Group F (4 assertions), 28 in the file
  • bash tests/security/test_error_msg_leaks.sh — 874 checks, 0 failures
  • Tested manually in the REPL — n/a, reachable only at report time

Reproduced in miniature — a polyglot script under a quality gate that always trips:

GovernanceCheck chained RunEnd declares verifier
pre-fix 10 (5 duplicated) 13 8 BREAK
post-fix 5 8 8 verified clean

Vacuity-checked: reverting the fix fails F-02, F-03 and F-04, with F-01 still passing. F-01 is the control — unless the quality gate actually fires, writeReports ran once and the other three prove nothing.

Related Issues

Follow-up to #109. Same defect class — chained events written without the count keeping up — found one wrong conclusion apart; docs/governance-campaign-findings.md records both, and corrects my earlier claim.


Generated by Claude Code

A keyed run showed L24-06 still failing after #109:

  BREAK in run ...: RunEnd declares 737 chained events but 785 observed

#109 was confirmed by that run — every small run group came back diff 0,
exactly where the end-of-run health warnings fire. But I called that writer the
complete explanation, and it was not. Every large run in the earlier sample
balanced exactly, which is consistent with one writer at unit 2 — only because
none of those runs tripped the quality gate. The second cause was invisible in
that sample, not absent.

writeReports() is called from ~17 sites and a clean execute() is not the last
of them: main.cpp's contract-error, quality-gate and baseline-regression exits
all call it again after the VM has already written and sealed the run. The five
file report formats are idempotent because they truncate and rewrite, but
telemetry appends and check_results_ is never cleared, so the second call
re-emitted the entire set after RunEnd had declared the count, and
run_end_emitted_ stopped the declaration ever catching up. The residual is one
full second dump: GovernanceCheck 38 + RuleViolation 8 + summary 1 + snapshot 1.
The double-call was already known — the comment "writeReports() already called
inside execute() on success" sits three lines above two more sites that do it.

Fixed at the invariant rather than the instance. RunEnd re-emits whenever
chained events followed the previous anchor; the verifier reads the last RunEnd
per run_id, so the newest declaration reconciles, and any future writer that
appends after an anchor is self-healing. Separately the dump resumes from
telemetry_results_dumped_ instead of restarting, the dedup key set persists
across calls so the guarantee still holds, and ScoringSnapshot re-emits only
when the score moved.

The duplicate records mattered beyond the verifier: every count derived from the
telemetry file was inflated, including L25-03's taint total, which had been
reading roughly double. Attributed by source location the same run has 8
distinct sink sites and 7-8 violations per segment, not the 4 sites that comment
claimed. The baseline is left at 18 as a loose ceiling and marked for
re-measurement on a keyed run built after this fix, since only a run can say
what the corrected count is.

Reproduced in miniature: pre-fix 10 records / 13 chained / declares 8 / BREAK,
post-fix 5 / 8 / 8 / clean. Group F asserts it, and F-01 is the control —
unless the quality gate actually fires, writeReports ran once and F-02..F-04
prove nothing. Reverting the fix fails F-02, F-03 and F-04 with the control
still passing.

Full suite: 441 tests, 0 unexpected failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

The post-fix keyed run settled what two data points could not. L25-03 measured
7, 8, 18, then 8 — and the 18 is the single run where the writeReports
double-dump fired, counting most violations twice. The apparent run-to-run
variance was that artifact, not behaviour: a violation fires about once per
distinct sink site, so the total tracks the ~8 sites rather than the length of
the run.

The earlier comment reasoned the count "is NOT monotone in run length" and
declined to tighten on that basis. Declining on two points was right; the
premise was reading the artifact as signal.

Inflation is now impossible, so 18 -> 12: ~50% headroom over every
non-inflated measurement, while still catching a leak. The build path runs 16
extraction sites across every feature iteration, so losing its sanitizer adds
far more than the 4 violations of slack.

Also records the run that confirms the reconciliation fix live — all four run
groups at diff 0, including a 708-event group carrying two RunEnd anchors, the
second declaring 708 of 708. That group reached its second writeReports()
through an exit path other than the quality gate, so the invariant held
somewhere the constructed repro never went.

Replayed the assertion: 8 passes, 13 fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@b-macker
b-macker marked this pull request as ready for review August 1, 2026 16:36
@b-macker
b-macker merged commit 1b4a8de into master Aug 1, 2026
26 of 28 checks passed
@b-macker
b-macker deleted the claude/naab-inadmissible-action-prevention-4cmn1m branch August 1, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants