A CLI built as a stateless verification-loop — sanity check on the 12-factor mapping?

[HumanBean17]

Hi — I'm building agctl. It's a CLI for verifying distributed systems (HTTP, Kafka, DBs, gRPC, logs) plus a built-in mock server. I want to be upfront about what it is not: it's not valuable because it can reach a Kafka broker or a Postgres instance — an agent can already do that with kcat / psql / an MCP tool. It's valuable because it makes that verification declarative, reproducible, and evidence-producing, and it handles the one part agents genuinely struggle to improvise well: a mock that's observable and fail-loud.

I built it before I'd read the 12 Factors end-to-end; when I did, I realized it lands on a chunk of them — and since the verification-loop pillar is the exact problem it targets, I wanted to put it here for feedback rather than quietly ship it.

The four things I think are the actual differentiators:

1. Templates — the contract is checked into the repo, not re-derived every session. Every HTTP call, SQL query, Kafka pattern, gRPC call, and log source is a named, parameterized template in agctl.yaml. The agent doesn't re-discover endpoints, auth headers, or SQL each run (which is exactly where it drifts/hallucinates); it fills parameters via --param against a human-authored, reviewable contract. agctl discover lists them; agctl config validate fails a bad template loud at load, not at runtime. → F4 (tools = structured outputs) + F3 (own your context window — the contract lives outside the model, it isn't re-derived each turn).

2. Runbooks — verification embedded in the plan, not bolted on after. A runbook is pure markdown that carries its own fixtures as a co-located overlay (<runbook>.agctl.yaml) and a Preconditions line. The workflow that worked on my brownfield project: the agent writes an implementation plan that embeds the runbook and the exact verification commands up front, then executes them and returns the JSON evidence — actual DB rows, Kafka messages, HTTP results, log entries — so a human confirms "this works" without re-running anything. The runbook is the deterministic back-pressure: same commands, same fixtures, same verdict, every time. → the verification-loop / back-pressure pillar.

3. The embedded mock — declarative, observable, fail-loud. agctl mock serves HTTP stubs, Kafka reactors, and gRPC stubs from the same agctl.yaml. The objection writes itself: an agent can reproduce any behavior by itself — hand-roll a Flask stub, write a Kafka consumer script. The reason I still built this: that improvisation is exactly where false-greens come from. agctl's mock is declarative (no script to get subtly wrong) and emits a structured NDJSON event stream — http.hit / http.unmatched / kafka.reacted / kafka.error / grpc.unmatched / capture.missing — so "the SUT sent the right thing to the right stub" becomes a line of JSON, and a plausible-but-wrong canned response (the classic mock failure mode) surfaces as a structured event + exit code instead of passing silently. → F9 (compact errors into context — the agent reads one JSON line, it doesn't grep a log).

4. Logs as a first-class evidence channel. (I'd left this out before.) agctl logs query/assert/tail normalizes sources into a canonical entry model, so logs are an assertion target, not "go read the file." The one I keep reaching for is the negative assertion: logs assert --not fails if an ERROR or stack trace did appear in the window — "the change didn't break anything loudly" as a deterministic check.

The mapping, tightened:

agctl differentiator	Factor
Templates checked into the repo; agent fills params against a reviewable contract	F4 + F3
Runbook + overlay carries fixtures + verification; agent returns JSON evidence	verification-loop / back-pressure
Mock failure-stream surfaces unmatched / errors / capture-misses as structured events + exit codes	F9
One JSON per call · deterministic exit 0/1/2 · stateless	F12 (stateless reducer)

The case study that made me post. I drove it against a ~7-year Spring + Kafka + Postgres system with no test harness. The agent authored the agctl.yaml contract (templates for the HTTP API, the SQL, the Kafka topics, the log sources), wrote the runbook into the plan, executed it, and handed back the evidence bundle — including the mock's failure-stream showing every expected stub was hit and nothing unmatched.

What I'd value feedback on:

• Does the "declarative templates + runbook + observable mock" framing hold up as the differentiator, or is "just let the agent call the infra" enough in practice?
• On the mock specifically — is a declarative/observable mock interesting here, or do people mostly let the agent hand-roll stubs? (genuinely asking — I've only stress-tested my own workflow.)
• Anything in the F9 direction (self-debugging failures) you'd do differently?

Repo: https://github.com/HumanBean17/agctl.

Not here to self-promote — genuinely want a sanity check from people who think about this layer full-time.

Note: drafted and reviewed by me; written and polished with Claude.