[AAASM-5529] ✅ (tests): Add enforcement-truth negative controls to the quick-start

[Chisanan232]

Description

Adds enforcement-truth negative controls to the documented Python quick-start path (AAASM-5529, Epic AAASM-5526, Goal CBLPCRLM-13).

A negative control proves a denial prevented a side effect, not that a DENIED message was produced. Every existing deny test in this repo asserts either the returned verdict dict or an empty executed list captured by a closure. Both show the SDK did not call a function it holds a reference to; neither shows that the effect the tool exists to produce did not happen.

Each control here:

1. runs the real init_assembly so the interceptor under test is the RuntimeQueryInterceptor the SDK actually builds (build_governance_interceptor is wrapped by a call-through spy, not replaced, so the real _register_adapters runs);
2. drives the SDK's own governed-call chain (agent_assembly/adapters/_shared/tool_governance.py::run_governed_async_tool, the shared pre-execution gate behind the Google ADK and Pydantic AI quick-start tabs) so the SDK, not the test, decides whether the tool body runs;
3. asserts a real side effect — a file on disk, an HTTP request delivered to a live loopback listener — present under allow and absent under deny.

The side-effect assertion runs before the exception assertion on purpose. Asserting the exception first would short-circuit when enforcement is removed, leaving the absence check unexercised — the falsification run would then only prove "no error was raised".

Corrections to the first revision of this description

An independent review returned CHANGES-REQUIRED on three points. All three are recorded here rather than silently overwritten.

1. One control did not actually follow the ordering rule stated above. TestDenyIsAttributable::test_the_runtime_saw_the_agent_and_tool_the_deny_was_decided_against asserted isinstance(outcome, ToolExecutionBlockedError) before assert file_effect.occurred() is False. Under the falsification mutation that assert aborted the test first, so that control's absence check had never been shown to bite — it was passing for the weak reason this suite exists to eliminate. The other three deny controls were already correct. Fixed in a4027f2 by swapping the two lines to match the shape the siblings already use. Before/after evidence below.

The evidence of this was visible in the first revision of this description and was missed: the falsification block summarised 4 failed but quoted only 2 assertion messages, and the omitted pair contained the isinstance failure that contradicts the sentence immediately beneath it. A quoted failure list shorter than the failure count is not evidence for a claim about every control.

2. The mypy numbers were wrong. The first revision claimed "exit 1, 9 errors, identical set to the baseline (9 errors / 186 files vs 9 errors / 188 files)". Re-measured with a cleared .mypy_cache on both sides, running the exact pre-commit gate, the real figures are branch: exit 0, 0 errors, 188 source files and pristine main (a838c13): exit 1, 3 errors in 1 file, 186 source files. The substantive conclusion — zero new type errors introduced — holds and is in fact stronger than claimed, but "9 errors on the branch" was never reproducible.

3. The bench exculpation was stated more strongly than the evidence supports. The first revision said both latency failures "reproduce running test/bench/ alone", implying determinism. They are flaky, not deterministic: test/bench alone reproduced them in 2 of 3 runs and was green in the third. The non-attribution conclusion still holds, and is now backed by the measurement that was missing — the same test fails on pristine main with none of this PR's files present. Numbers below.

Falsification evidence

Produced by removing the deny in the SDK (not in the test) and re-running the whole control file:

# agent_assembly/adapters/_shared/tool_governance.py:225
-    if status != "allow":
+    if False:  # enforcement removed

Before a4027f2 — 4 failed, 5 passed, 4 rerun (exit 1). Three controls failed on the side effect; one did not:

test_negative_control_denied_write_leaves_no_file
>       assert file_effect.occurred() is False
E       AssertionError: assert True is False

test_negative_control_denied_egress_never_reaches_the_listener
>       assert network_effect.occurred() is False
E       AssertionError: assert True is False

test_the_runtime_saw_the_agent_and_tool_the_deny_was_decided_against     <-- the defect
>       assert isinstance(outcome, ToolExecutionBlockedError)
E       AssertionError: assert False
E        +  where False = isinstance('/private/var/.../denied-write.txt', ToolExecutionBlockedError)

test_an_unavailable_native_runtime_denies_rather_than_silently_allowing
>       assert file_effect.occurred() is False
E       AssertionError: assert True is False

After a4027f2 — 4 failed, 5 passed, 4 rerun (exit 1). All four now fail on the absence assertion, and the third control's message has genuinely changed:

test_the_runtime_saw_the_agent_and_tool_the_deny_was_decided_against
>       assert file_effect.occurred() is False
E       AssertionError: assert True is False
E        +  where True = occurred()
E        +    where occurred = FileSideEffect(path=PosixPath('/private/var/.../denied-write.txt')).occurred

Grepping the after-run output for isinstance(outcome returns 0 hits against a positive control of 4 hits for occurred() is False in the same file, so the empty result is a real absence and not a broken probe.

Source restored after each run in the same uninterruptible sequence; verified by git status --porcelain being empty and a positive grep confirming if status != "allow": is back at line 225, not merely that the mutation marker is gone.

Type of Change

• ✨ New feature (test coverage)
• 🔧 Bug fix (unrelated pre-existing gate repair, see below)

Breaking Changes

• No

Related Issues

• Related JIRA ticket: AAASM-5529
• Parent Epic: AAASM-5526 — Host-wide capability mediation and truthful governance guarantees
• Goal: CBLPCRLM-13
• Blocks: AAASM-5589 — Embed an evidence-backed denied-action proof into the public website
• Coordinates with: AAASM-4991 (Node default path), AAASM-5016, AAASM-4452, AAASM-5527

Sibling PRs (same ticket, one per SDK)

Repo	PR
node-sdk	ai-agent-assembly/node-sdk#352
go-sdk	ai-agent-assembly/go-sdk#194

Recommended merge order: python-sdk → go-sdk → node-sdk. They are independent (no shared code), so the order only reflects reviewer load: this one and go-sdk are pure additions, while the node-sdk PR carries the findings that need a decision.

Acceptance-criteria mapping

AC	Where
Python quick-start proves allow and deny side effects end to end	TestFilesystemSideEffect, TestNetworkSideEffect
Each deny test asserts the tool body was not executed, not only that an error was logged	occurred() / content() / listener request log; asserted before the exception in all four deny controls (see correction 1)
Each test verifies agent/tool identity in audit evidence	TestDenyIsAttributable — identity read off the query the SDK presented to the authoritative runtime, plus the allow path's record_result
No-op / missing-runtime paths cannot display a protected/enforced success state	TestDegradedRuntimeCannotLookProtected
Supported adapter/tool path actually invokes RuntimeQueryInterceptor	The spy asserts init_assembly built one; the chain calls its check_tool_start
Existing AAASM-4991 / AAASM-5016 reused or linked, not duplicated	Fixture reuses test/unit/core/_fake_core.py; no adapter code touched

Not covered here (reported to the ticket owner, not silently dropped): the CI job running the quick-start from a clean environment, and doc-snippet drift gating of the new controls.

Testing

• Unit tests added
• All tests passing

How to read the pass counts below: pytest.ini sets --reruns 1, so every quoted count is pass-after-retry — a test that failed once and passed on the retry is reported as passed, with a rerun in the summary line. Also, -p no:randomly (used in the first revision's commands) is inert in this repo: pytest-randomly is not installed, so it disables nothing and ordering is plain collection order.

All measured at a4027f2 on macOS 15 / Python 3.13.3 / mypy 2.2.0.

Gate	Command	Exit
Target suite	.venv/bin/python -m pytest test/unit/test_quickstart_negative_control.py --no-cov -q	0 — 9 passed
Full suite	.venv/bin/python -m pytest test/ --no-cov -q	0 — 1208 passed, 16 skipped, 1 rerun
Full suite minus benchmarks	.venv/bin/python -m pytest test/ --ignore=test/bench --no-cov -q	0 — 1179 passed, 14 skipped, 1 rerun
Lint	.venv/bin/ruff check .	0 — all checks passed
Format (PR's files)	.venv/bin/ruff format --check on the 3 changed files	0 — 3 files already formatted
Format (repo-wide)	.venv/bin/ruff format --check .	1 — scripts/check_contact_metadata.py would be reformatted. Pre-existing; fails identically on main, untouched here.
Types	uv run mypy --ignore-missing-imports --show-traceback (the exact pre-commit hook), .mypy_cache cleared	0 — Success: no issues found in 188 source files
Types, baseline	same command on pristine main a838c13, .mypy_cache cleared	1 — Found 3 errors in 1 file (checked 186 source files), all three in test_runner_spawn_patch.py
Pre-commit	uv run pre-commit run --files <the 3 changed files>	0 — every hook Passed (ruff check, ruff format, mypy, …)

The branch's type-error set is empty, so it is trivially a strict subset of main's three. The three baseline errors are exactly the ones the included fix below removes.

Benchmark flakiness (not attributable to this PR)

test/bench/test_latency_contracts.py asserts wall-clock latency budgets and is timing-flaky on a loaded workstation. Measured three consecutive runs on each side, pytest test/bench --no-cov -q:

Tree	Run 1	Run 2	Run 3
This branch a4027f2	exit 1 — 2 failed, 27 passed, 2 skipped	exit 1 — 2 failed, 27 passed, 2 skipped	exit 0 — 29 passed, 2 skipped
Pristine main a838c13	exit 1 — 1 failed, 28 passed, 2 skipped	exit 1 — 1 failed, 28 passed, 2 skipped	exit 0 — 29 passed, 2 skipped

test_init_assembly_coldstart_latency fails on pristine main, with none of this PR's files collected — so the failure class predates this branch. test_detection_latency_under_50ms also tripped on the branch runs; both are wall-clock budget assertions on a machine under load, and neither timing is cited as evidence anywhere in this PR. The full-suite run above happened to be green on all of it.

Included unrelated fix (called out, not hidden)

🚨 (test): Mark patched Runner.run stubs positional-only for mypy 2.2 — mypy 2.2.0 checks classmethod()'s argument against def (type[Never], /, ...), so a named first parameter is rejected. The pre-commit mypy hook fails on a clean remote/main with exactly these 3 errors (measured above), which blocks every commit in this repo (--no-verify is not an option). The / marker is runtime-inert: every call site passes the agent positionally, and inspect.signature still yields ['agent', 'input', 'kwargs']. Happy to split this into its own PR if preferred.

Checklist

• Code follows project style guidelines
• Self-review completed
• Comments explain the WHY (why a spy is insufficient, why assertion order matters)
• Documentation updated if needed — none required; no public surface changed
• All tests passing (see table)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud