Skip to content

Govern the example's LLM-output-to-execution flow - #108

Merged
b-macker merged 2 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m
Jul 31, 2026
Merged

Govern the example's LLM-output-to-execution flow#108
b-macker merged 2 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m

Conversation

@b-macker

@b-macker b-macker commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Summary

living-script_extended takes LLM output, writes it to pipeline.py, and executes it via codegen. That is the most dangerous flow in the example and it was ungoverned — no taint_tracking section, so the default enabled: false applied. agent.send() already marks its return tainted (agent_impl.cpp:4560): the source was wired, the config was off.

This adds taint_tracking (advisory) with a sanitize_llm_code() trust boundary, integrity.blocked_flags as a deliberate negative-control probe, and Levels 25/26 that assert both actually fired.

What the live keyed run established

The sanitizer boundary works. All 16 build-path extraction sites produced zero violations. L25-01/02 and L26-01/02/03 passed, no build-path regressions, all 22 L4 phase markers passed.

L25-03 failed — correctly — and the assertion was wrong, not the code. It asserted a total under soft_after (8); the run produced 18. 17 of those come from four non-build-path sites that write agent-derived data without passing through extraction:

site n assessment
validate_python_tool() 6 Writes unvalidated model code to _tool_check.py, then AST-parses it. The write precedes validation, so calling it sanitized would be false. True positive.
operator config writes 6 LLM JSON parsed and field-validated before re-serialising. A validate_-RHS binding would be honest and would lower the baseline — left for its own change.
memory persistence 2 Aggregate tracking data, transitively tainted.
unlocated (line 0) 4 Dynamic/codegen writes with no source location.

The fix, and what it can honestly claim

The assertion I wanted — "no violation targets pipeline.py/models.py/test_pipeline.py" — is not expressible. The violation message (governance_taint.cpp:270) names the sink type and the source file:line, never the write target; RuleViolation records carry the source location too. Attribution is possible only by source line, and line numbers move on any edit.

So L25-03 now detects growth against a documented baseline. The build path contributes zero; a leak there adds violations, raises the total, and fails. Each known site is enumerated in the code with why it is expected, and raising the number is a reviewed decision rather than a routine edit.

Two corrections to earlier claims in this PR

  • The escalation I predicted never fired. The run recorded zero ESCALATED lines with the occurrence counter peaking at 5/8, because epoch boundaries reset advisory history. My earlier rationale — that an unsanitized build path "would harden into a SOFT block and the build would stop producing" — was not borne out at 18 violations. soft_after is demoted to a printed note rather than the pass condition, and the comments are corrected. The sanitizer boundary stays because it is the honest model of where model output becomes a checked artifact, not because it averts breakage.
  • validate_python_tool firing 6 times is correct behaviour, not a naming miss. checkRhsSanitized() clears taint on the assignment RHS; being inside a function whose name matches a sanitizer prefix does nothing for values it writes internally. Now recorded in the code, since it is a natural mistake to make.

temporal_coupling deliberately NOT added

It detects inter-agent timing correlation, but this config pins max_concurrent: 1 / pool_size: 1 and sets delay_between_calls_ms: 1000 on all 7 agents. Calls are correlated by configuration, so the signal would measure the dispatch settings rather than agent behaviour — a misleading signal in an artifact whose purpose is to be believable evidence.

Changes

  • src/govern.jsontaint_tracking (advisory, with sanitizers) and integrity.blocked_flags. 12 added lines, no reformatting.
  • src/living-script.naabsanitize_llm_code() boundary with the RHS mechanic and known-site inventory documented, 16 extraction sites routed through it, new TAINT_AUDIT and INTEGRITY_PROBE phases.
  • run.sh — Levels 25 and 26, baseline-based L25-03, phases wired into the tracking loop.

Test Plan

  • bash run-all-tests.sh441 tests, 0 unexpected failures
  • Stub-verified without live keys: unsanitized control write → 1 taint_tracking.sink_violation, build path → 0; blocked flag → exit 3 not executed, control run → exit 0 executed
  • Live keyed run: L25-01/02 and L26-01/02/03 pass; sanitizer boundary confirmed at zero across all 16 build-path sites; L25-03 surfaced the four uncovered sites
  • L25-03 replayed against the observed numbers: 18 passes, 19 and 25 fail, stub's 1 still passes — the level accepts the known state and rejects growth
  • parse clean; check diagnostics unchanged at 35; run.sh passes bash -n; govern.json valid

Known, out of scope

  • L24-06-telemetry failed in the same run: "RunEnd declares 717 chained events but 770 observed" — 53 chained events written without incrementing chained_events_this_run_. That is Attribute attestation signatures, and prove the evidence layer fires #106's verifier catching a real evidence-integrity bug. Different subsystem; it gets its own traced investigation rather than riding along here.
  • RUN-02 (feature 2 incomplete, pytest kept failing) — runtime quality of the generated code, unrelated to this PR.
  • Routing the operator-config and memory writes through validate_-RHS bindings, which would lower the baseline.

living-script_extended takes LLM output, writes it to pipeline.py and executes
it via codegen. That is the most dangerous flow in the example and it was
ungoverned: no taint_tracking section, so the default enabled:false applied.
agent.send() already marks its return tainted — the source was wired, the
config was off.

Enabling it at "advisory" was not safe, which only surfaced by tracing the
interaction rather than assuming additivity. advisory_escalation is enabled
with soft_after 8, and escalation counts per rule_name
(emitted_advisories_[rule_name]); this script writes LLM-derived files far more
than eight times. The advisory would have hardened into a SOFT block partway
through and the pipeline build would have stopped producing. Note
codegen.allow_tainted_code was already true, so the author had considered
tainted code reaching codegen and permitted it deliberately.

So the flow is modelled honestly instead: sanitize_llm_code() wraps
agent.extract_code() and all sixteen extraction sites route through it.
Extraction is where model output stops being an opaque response and becomes a
checked artifact, which makes it the truthful place to clear taint. The
sanitize_ prefix is what taint_tracking.sanitizers matches on, so the name is
load-bearing.

A clean violation count is indistinguishable from taint being switched off, so
the phase makes one deliberate unsanitized write as a positive control. Two
things had to be right for it to work, both found empirically: the tainted
value must be bound to a variable, since taint is name-based and an inline
expression carries none; and the count belongs in run.sh after the run, because
taint violations are RuleViolation records written by writeTelemetry() — a bulk
dump at shutdown — not the per-event writeAgentTelemetry() path the evidence
phase counts. An in-script count reads zero however many fired.

L25-03 bounds the build path against soft_after read from govern.json rather
than an arbitrary number, so the assertion means "the sanitizer is still doing
its job" rather than "the count looks small".

integrity.blocked_flags is inert as config since run.sh never passes those
flags, so the probe invokes the binary WITH one and asserts refusal, plus a
control run without it — otherwise a refusal would be indistinguishable from a
broken binary or path.

temporal_coupling is deliberately NOT added. It detects inter-agent timing
correlation, but this config pins max_concurrent and pool_size to 1 and sets
delay_between_calls_ms to 1000 on all seven agents, so calls are correlated by
configuration. The signal would measure the dispatch settings rather than agent
behaviour — a misleading signal in an artifact whose purpose is to be
believable evidence.

Verified against the stub without live keys: unsanitized control write produces
1 taint sink violation, build path stays at 0, blocked flag exits 3 without
executing while the control run exits 0 and executes.

Full suite: 441 tests, 0 unexpected failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@github-actions

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

The live keyed run failed L25-03 with 18 violations against an asserted
ceiling of soft_after (8). The failure was correct and the assertion was
wrong.

What the run established: the sanitizer boundary works. All 16
sanitize_llm_code() build-path extraction sites produced zero violations, and
no build-path phase regressed. 17 of the 18 violations come from four
non-build-path sites that write agent-derived data to files without passing
through extraction at all — a tool callback, two operator-config writers,
memory persistence, plus four writes with no source location. I asserted a
total on the assumption the build path was the only route from agent data to a
file sink. It is not.

The assertion I wanted — "no violation targets pipeline.py, models.py or
test_pipeline.py" — is not expressible. The violation message names the sink
TYPE and the SOURCE file:line, never the write target, and the RuleViolation
record's file/line are the source location too. Attribution is possible only by
source line, and line numbers move on any edit. So L25-03 now detects GROWTH
against a documented baseline: the build path contributes zero, a leak there
adds violations, the total rises, and the level fails. Each of the four known
sites is enumerated with why it is expected, and raising the number is a
reviewed decision rather than a routine edit.

soft_after is demoted to a printed note. The run recorded ZERO escalations with
the occurrence counter peaking at 5/8, because epoch boundaries reset advisory
history — so a total above soft_after does not by itself mean the advisory
hardened into a block. My earlier claim that an unsanitized build path "would
harden into a SOFT block and the build would stop producing" was not borne out
at 18 violations, and the comments saying so are corrected. The sanitizer
boundary stays because it is the honest model of where model output becomes a
checked artifact, not because it averts breakage.

Also recorded: checkRhsSanitized() clears taint on the ASSIGNMENT RHS, so being
inside a function whose name matches a sanitizer prefix does nothing for values
that function writes internally. That is precisely why validate_python_tool()
trips the sink check despite its name — it writes its tainted argument
directly. Correct behaviour, and a natural mistake to make.

Left deliberately undone: routing the operator-config and memory writes through
validate_-RHS bindings would be honest and would lower the baseline, but doing
it in the same commit that establishes the baseline makes the assertion harder
to review.

Verified by replaying the observed numbers: 18 passes, 19 and 25 fail, and the
stub's 1 still passes.

Full suite: 441 tests, 0 unexpected failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@b-macker
b-macker marked this pull request as ready for review July 31, 2026 10:17
@b-macker
b-macker merged commit 142c015 into master Jul 31, 2026
23 checks passed
@b-macker
b-macker deleted the claude/naab-inadmissible-action-prevention-4cmn1m branch July 31, 2026 10:17
b-macker added a commit that referenced this pull request Jul 31, 2026
)

L24-06 — the level added in #106 precisely because the example had been
building a tamper-evident chain nothing ever verified — failed on real run
files: "RunEnd declares 717 chained events but 770 observed". That is a hard
BREAK, exit 1, on files nobody had touched.

emitEndOfRunHealthWarnings() is the only chained-telemetry writer in the
engine that did not go through chainPrevLocked(). It seeded prev_hash from
the in-memory last_telemetry_hash_ and never incremented
chained_events_this_run_. Three symptoms from those two lines:

  - RunEnd under-declared its count, so the verifier reported tampering.
  - When the warnings were a process's first chained events the in-memory
    hash was empty, so prev_hash fell back to genesis mid-file — a spurious
    LEGACY RESTART on any shared telemetry file.
  - The lazy RunStart anchor landed behind the events it anchors, because
    only chainPrevLocked() emits it.

Reproduced on a print("hello") script with governance_health, CDD and BSD
enabled and no agent activity: 4 chained events, RunEnd declaring 2. The
unit is 2 per affected run — the two inert-instrumentation warnings — which
matches the diff observed across four live run groups. The originally
reported 53 was an artifact of aggregating run groups, not one large gap.

A verifier that cries tamper on its own output is worse than no verifier: it
trains the reader to ignore the one signal meant to be unignorable.

The fix is what every other chained writer already does — chainPrevLocked(fp)
plus the increment, inside the lock the lambda already held. Group E covers
it, and E-01 is the control: without health warnings actually firing, E-02..
E-04 would be vacuous. Verified by reverting the fix and confirming all three
fail while E-01 still passes; Group A passes either way, which is why this
survived.

Also here:

docs/governance-campaign-findings.md gains a transition-admissibility phase
section covering the five defects from #104-#108 plus this one, each with
mechanism, pinning test, and a separately-scoped live status. A fourth
method note records the defect class this phase kept producing — an
assertion that looks specific but is satisfiable without the property
holding (C-05 matching an empty value, L24-02's unsound equality, L25-03's
wrong-scope total). Running the degraded case was the only defence that ever
caught it.

L25-03's baseline comment records the second keyed observation: 7-8 against
a ceiling of 18, from a more complete run than the one that produced 18. The
count is not monotone in run length, so two non-monotone points do not
support tightening — a baseline that fails on a healthy run teaches everyone
to raise it. Left at 18 with the reasoning written down; tightening needs the
variance attributed to a site first.

Full suite: 441 tests, 0 unexpected failures. Security leak check: 874/0.


Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants