Govern the example's LLM-output-to-execution flow - #108
Merged
b-macker merged 2 commits intoJul 31, 2026
Conversation
living-script_extended takes LLM output, writes it to pipeline.py and executes it via codegen. That is the most dangerous flow in the example and it was ungoverned: no taint_tracking section, so the default enabled:false applied. agent.send() already marks its return tainted — the source was wired, the config was off. Enabling it at "advisory" was not safe, which only surfaced by tracing the interaction rather than assuming additivity. advisory_escalation is enabled with soft_after 8, and escalation counts per rule_name (emitted_advisories_[rule_name]); this script writes LLM-derived files far more than eight times. The advisory would have hardened into a SOFT block partway through and the pipeline build would have stopped producing. Note codegen.allow_tainted_code was already true, so the author had considered tainted code reaching codegen and permitted it deliberately. So the flow is modelled honestly instead: sanitize_llm_code() wraps agent.extract_code() and all sixteen extraction sites route through it. Extraction is where model output stops being an opaque response and becomes a checked artifact, which makes it the truthful place to clear taint. The sanitize_ prefix is what taint_tracking.sanitizers matches on, so the name is load-bearing. A clean violation count is indistinguishable from taint being switched off, so the phase makes one deliberate unsanitized write as a positive control. Two things had to be right for it to work, both found empirically: the tainted value must be bound to a variable, since taint is name-based and an inline expression carries none; and the count belongs in run.sh after the run, because taint violations are RuleViolation records written by writeTelemetry() — a bulk dump at shutdown — not the per-event writeAgentTelemetry() path the evidence phase counts. An in-script count reads zero however many fired. L25-03 bounds the build path against soft_after read from govern.json rather than an arbitrary number, so the assertion means "the sanitizer is still doing its job" rather than "the count looks small". integrity.blocked_flags is inert as config since run.sh never passes those flags, so the probe invokes the binary WITH one and asserts refusal, plus a control run without it — otherwise a refusal would be indistinguishable from a broken binary or path. temporal_coupling is deliberately NOT added. It detects inter-agent timing correlation, but this config pins max_concurrent and pool_size to 1 and sets delay_between_calls_ms to 1000 on all seven agents, so calls are correlated by configuration. The signal would measure the dispatch settings rather than agent behaviour — a misleading signal in an artifact whose purpose is to be believable evidence. Verified against the stub without live keys: unsanitized control write produces 1 taint sink violation, build path stays at 0, blocked flag exits 3 without executing while the control run exits 0 and executes. Full suite: 441 tests, 0 unexpected failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
NAAb Governance Report
All governance checks passed! Generated by NAAb Governance Engine v4.0 |
The live keyed run failed L25-03 with 18 violations against an asserted ceiling of soft_after (8). The failure was correct and the assertion was wrong. What the run established: the sanitizer boundary works. All 16 sanitize_llm_code() build-path extraction sites produced zero violations, and no build-path phase regressed. 17 of the 18 violations come from four non-build-path sites that write agent-derived data to files without passing through extraction at all — a tool callback, two operator-config writers, memory persistence, plus four writes with no source location. I asserted a total on the assumption the build path was the only route from agent data to a file sink. It is not. The assertion I wanted — "no violation targets pipeline.py, models.py or test_pipeline.py" — is not expressible. The violation message names the sink TYPE and the SOURCE file:line, never the write target, and the RuleViolation record's file/line are the source location too. Attribution is possible only by source line, and line numbers move on any edit. So L25-03 now detects GROWTH against a documented baseline: the build path contributes zero, a leak there adds violations, the total rises, and the level fails. Each of the four known sites is enumerated with why it is expected, and raising the number is a reviewed decision rather than a routine edit. soft_after is demoted to a printed note. The run recorded ZERO escalations with the occurrence counter peaking at 5/8, because epoch boundaries reset advisory history — so a total above soft_after does not by itself mean the advisory hardened into a block. My earlier claim that an unsanitized build path "would harden into a SOFT block and the build would stop producing" was not borne out at 18 violations, and the comments saying so are corrected. The sanitizer boundary stays because it is the honest model of where model output becomes a checked artifact, not because it averts breakage. Also recorded: checkRhsSanitized() clears taint on the ASSIGNMENT RHS, so being inside a function whose name matches a sanitizer prefix does nothing for values that function writes internally. That is precisely why validate_python_tool() trips the sink check despite its name — it writes its tainted argument directly. Correct behaviour, and a natural mistake to make. Left deliberately undone: routing the operator-config and memory writes through validate_-RHS bindings would be honest and would lower the baseline, but doing it in the same commit that establishes the baseline makes the assertion harder to review. Verified by replaying the observed numbers: 18 passes, 19 and 25 fail, and the stub's 1 still passes. Full suite: 441 tests, 0 unexpected failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
b-macker
marked this pull request as ready for review
July 31, 2026 10:17
5 tasks
b-macker
added a commit
that referenced
this pull request
Jul 31, 2026
) L24-06 — the level added in #106 precisely because the example had been building a tamper-evident chain nothing ever verified — failed on real run files: "RunEnd declares 717 chained events but 770 observed". That is a hard BREAK, exit 1, on files nobody had touched. emitEndOfRunHealthWarnings() is the only chained-telemetry writer in the engine that did not go through chainPrevLocked(). It seeded prev_hash from the in-memory last_telemetry_hash_ and never incremented chained_events_this_run_. Three symptoms from those two lines: - RunEnd under-declared its count, so the verifier reported tampering. - When the warnings were a process's first chained events the in-memory hash was empty, so prev_hash fell back to genesis mid-file — a spurious LEGACY RESTART on any shared telemetry file. - The lazy RunStart anchor landed behind the events it anchors, because only chainPrevLocked() emits it. Reproduced on a print("hello") script with governance_health, CDD and BSD enabled and no agent activity: 4 chained events, RunEnd declaring 2. The unit is 2 per affected run — the two inert-instrumentation warnings — which matches the diff observed across four live run groups. The originally reported 53 was an artifact of aggregating run groups, not one large gap. A verifier that cries tamper on its own output is worse than no verifier: it trains the reader to ignore the one signal meant to be unignorable. The fix is what every other chained writer already does — chainPrevLocked(fp) plus the increment, inside the lock the lambda already held. Group E covers it, and E-01 is the control: without health warnings actually firing, E-02.. E-04 would be vacuous. Verified by reverting the fix and confirming all three fail while E-01 still passes; Group A passes either way, which is why this survived. Also here: docs/governance-campaign-findings.md gains a transition-admissibility phase section covering the five defects from #104-#108 plus this one, each with mechanism, pinning test, and a separately-scoped live status. A fourth method note records the defect class this phase kept producing — an assertion that looks specific but is satisfiable without the property holding (C-05 matching an empty value, L24-02's unsound equality, L25-03's wrong-scope total). Running the degraded case was the only defence that ever caught it. L25-03's baseline comment records the second keyed observation: 7-8 against a ceiling of 18, from a more complete run than the one that produced 18. The count is not monotone in run length, so two non-monotone points do not support tightening — a baseline that fails on a healthy run teaches everyone to raise it. Left at 18 with the reasoning written down; tightening needs the variance attributed to a site first. Full suite: 441 tests, 0 unexpected failures. Security leak check: 874/0. Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
living-script_extendedtakes LLM output, writes it topipeline.py, and executes it viacodegen. That is the most dangerous flow in the example and it was ungoverned — notaint_trackingsection, so the defaultenabled: falseapplied.agent.send()already marks its return tainted (agent_impl.cpp:4560): the source was wired, the config was off.This adds
taint_tracking(advisory) with asanitize_llm_code()trust boundary,integrity.blocked_flagsas a deliberate negative-control probe, and Levels 25/26 that assert both actually fired.What the live keyed run established
The sanitizer boundary works. All 16 build-path extraction sites produced zero violations. L25-01/02 and L26-01/02/03 passed, no build-path regressions, all 22 L4 phase markers passed.
L25-03 failed — correctly — and the assertion was wrong, not the code. It asserted a total under
soft_after(8); the run produced 18. 17 of those come from four non-build-path sites that write agent-derived data without passing through extraction:validate_python_tool()_tool_check.py, then AST-parses it. The write precedes validation, so calling it sanitized would be false. True positive.validate_-RHS binding would be honest and would lower the baseline — left for its own change.line 0)The fix, and what it can honestly claim
The assertion I wanted — "no violation targets
pipeline.py/models.py/test_pipeline.py" — is not expressible. The violation message (governance_taint.cpp:270) names the sink type and the sourcefile:line, never the write target;RuleViolationrecords carry the source location too. Attribution is possible only by source line, and line numbers move on any edit.So L25-03 now detects growth against a documented baseline. The build path contributes zero; a leak there adds violations, raises the total, and fails. Each known site is enumerated in the code with why it is expected, and raising the number is a reviewed decision rather than a routine edit.
Two corrections to earlier claims in this PR
ESCALATEDlines with the occurrence counter peaking at 5/8, because epoch boundaries reset advisory history. My earlier rationale — that an unsanitized build path "would harden into a SOFT block and the build would stop producing" — was not borne out at 18 violations.soft_afteris demoted to a printed note rather than the pass condition, and the comments are corrected. The sanitizer boundary stays because it is the honest model of where model output becomes a checked artifact, not because it averts breakage.validate_python_toolfiring 6 times is correct behaviour, not a naming miss.checkRhsSanitized()clears taint on the assignment RHS; being inside a function whose name matches a sanitizer prefix does nothing for values it writes internally. Now recorded in the code, since it is a natural mistake to make.temporal_coupling deliberately NOT added
It detects inter-agent timing correlation, but this config pins
max_concurrent: 1/pool_size: 1and setsdelay_between_calls_ms: 1000on all 7 agents. Calls are correlated by configuration, so the signal would measure the dispatch settings rather than agent behaviour — a misleading signal in an artifact whose purpose is to be believable evidence.Changes
src/govern.json—taint_tracking(advisory, with sanitizers) andintegrity.blocked_flags. 12 added lines, no reformatting.src/living-script.naab—sanitize_llm_code()boundary with the RHS mechanic and known-site inventory documented, 16 extraction sites routed through it, newTAINT_AUDITandINTEGRITY_PROBEphases.run.sh— Levels 25 and 26, baseline-based L25-03, phases wired into the tracking loop.Test Plan
bash run-all-tests.sh— 441 tests, 0 unexpected failurestaint_tracking.sink_violation, build path → 0; blocked flag → exit 3 not executed, control run → exit 0 executedparseclean;checkdiagnostics unchanged at 35;run.shpassesbash -n;govern.jsonvalidKnown, out of scope
L24-06-telemetryfailed in the same run: "RunEnd declares 717 chained events but 770 observed" — 53 chained events written without incrementingchained_events_this_run_. That is Attribute attestation signatures, and prove the evidence layer fires #106's verifier catching a real evidence-integrity bug. Different subsystem; it gets its own traced investigation rather than riding along here.RUN-02(feature 2 incomplete, pytest kept failing) — runtime quality of the generated code, unrelated to this PR.validate_-RHS bindings, which would lower the baseline.