Evidence

Evidence

A test suite and the examples check quality here, because the package makes behavioral claims and not statistical ones. The tests below ran on 2026-11-06, in the repository's own .venv and again in a fresh virtual environment built from a clean copy, on Windows with Python 3.12.10. Both runs gave the same counts.

Tests

$ .venv/Scripts/python -m pytest -q
........................................................................ [ 29%]
..........................................s............................. [ 59%]
........................................................................ [ 89%]
..........................                                               [100%]
241 passed, 1 skipped in 23.54s

File	Tests	Covers
tests/test_pool.py	72	The advisory pool: contracts, privacy, cloud rules, and result handling
tests/test_gateway_dispatch.py	43	Dispatch, refusals, handler states, and exit codes
tests/test_rules.py	37	Rule grammar, three-valued logic, and traces
tests/test_governance_units.py	34	Admission, budget, watchdog, loops, router, cache, and ledger
tests/test_governance_cli.py	23	The governance command line
tests/test_ollama_client.py	18	The Ollama client, through a real socket
tests/test_agent_registry.py	12	Agent definition files and their safety rules
tests/test_examples.py	2	The quick start and the console entry point
tests/test_live_ollama.py	1	A real model round trip, skipped by default

Two choices in this suite deserve attention. tests/fake_ollama.py runs a real HTTP server on a local port, so the tests reach the clients through a socket and not a mocked call, which covers request framing and status codes as well. The other choice concerns naming: tests such as test_a_handler_saying_passed_is_not_the_last_word, test_every_refusal_is_a_valid_result_packet, and test_a_model_cannot_relabel_its_own_result state a behavior in plain words, so the list of test names doubles as a specification.

What was measured

No field measurement exists. The repository records runs in tests and examples only, so it can support statements about behavior and none about savings. What follows is real output from commands run on 2026-11-06.

Admission refuses for different reasons, with different meanings. With stop_condition removed from examples/context_capsule.yaml:

{
  "status": "REQUIRE_REPAIR",
  "model": null,
  "reason": "missing stop condition"
}

With fresh_dynamic raised to 9,000 against a limit of 2,500:

{
  "status": "REJECT",
  "model": null,
  "reason": "fresh-token budget exceeded"
}

Both exit 1. The first says to fix the request, and the second says the request is too large.

A refusal is a valid packet, and unreadable input is a different case. A packet aimed at the unbuilt text.summarise returned blocked, with this entry in unresolved, and exited 1:

{'reason': 'capability_unbuilt', 'detail': "Capability 'text.summarise' declares no handler (intended executor: not written yet)"}

A file that is not JSON exited 2 instead:

task packet is not JSON: Expecting value: line 1 column 1 (char 0)

Repetition survives a changing timestamp. Three calls to check-loop-hook with the same tool and arguments and timestamps of 1, 2, and 3 returned plain allow twice, and the third added this warning while still exiting 0:

Repeated tool call detected (3x with identical arguments): Grep. Consider a different approach instead of repeating this call, or checkpoint if you are stuck.

Fallback depends on the reason and on the checkpoint. Run directly, choose_orchestrator() gave three distinct answers:

RuntimeError: Fallback blocked because no checkpoint exists
RouteDecision(model=<ModelRole.PRIMARY: 'primary'>, reason='failure is not fallback eligible', fallback_activated=False)
RouteDecision(model=<ModelRole.FALLBACK: 'fallback'>, reason='one-way fallback lock', fallback_activated=True)

Known failures and gaps

• The live round trip was not re-run for this page. The README records that tests/test_live_ollama.py passed once against a local Ollama model in 120 seconds. That is the only contact with a real model recorded in the repository, and it covers the pool alone.
• An empty rule context passes the shell gate. evaluate-rules with {} printed four unknowns and exited 0, which sits badly with the principle that missing data never looks like a pass.
• Unused declarations exist. REQUIRE_HUMAN, CircuitState, RunState, and ProgressProof have no code behind them, as Limits and Next Steps records.
• No test run covers Linux or macOS.

Check it yourself in 5 minutes

Everything here runs offline after installation. On Windows, use .venv\Scripts\ in place of .venv/bin/.

git clone https://github.com/dboles99/agent-gateway.git
cd agent-gateway
python -m venv .venv
.venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest -q                # 241 passed, 1 skipped
.venv/bin/python examples/quickstart.py      # the eight stages in How It Works

To see a refusal with its exit code, run this after the steps above:

.venv/bin/agent-gateway gateway capabilities --config examples/capabilities.yaml; echo "exit $?"
sed '/^stop_condition/d' examples/context_capsule.yaml > no-stop.yaml
.venv/bin/agent-gateway governance admit no-stop.yaml; echo "exit $?"

The first command prints 2 declared, 1 built, 1 unbuilt, 0 unresolvable and exit 0. The second prints REQUIRE_REPAIR and exit 1, which shows that the gate separates a capability nobody has built from a request nobody can finish.