Testing a Consumer

Testing a Consumer

A consumer of the runtime owns three things the runtime cannot test for it: its tools, its policy, and its agent. The runtime makes promises about all three, that nothing re-executes on its own after a crash, that a policy's answer decides what runs, that arguments are checked against the tool's schema first, and a consumer wants to see those promises hold for its own code, without a model and without a provider.

testkit is the core package for that. It is built on the public API and on the scripted package, adds no dependency (the schema library it fuzzes with is the one the core already requires), and is used from an ordinary Go test:

import "github.com/joeylking/agent-runtime/testkit"

It has four parts: an interruption harness that crashes the loop at a chosen point and resumes it, a policy conformance table, schema fuzzing of tool arguments, and a check that an agent renders the same model request every time. It shipped in v0.3.1. Since v0.5.0 it also has helpers that re-check a run under a policy. This page is about testing a consumer; Testing is about testing the runtime itself.

The interruption harness

testkit.Run runs a scenario in a fresh file-backed store, crashes it at each chosen point in turn, resumes it in a fresh driver each time, and asserts what the runtime promises across an interruption.

res := testkit.Run(t, testkit.Scenario{
    Tools:  tools,
    Policy: policy,
    Agent:  &scripted.Agent{Decisions: decisions},
}, testkit.Crash{Step: 0, At: testkit.AfterToolEffect})

The scenario

Field	Meaning
Tools, Policy	what a consumer hands NewDriver
Agent	usually a scripted.Agent or the consumer's own replayer; handed to every driver the harness builds, so it must decide from the recorded steps alone, as the runtime requires anyway. A replayer keyed by step index skips the decision of a step that was interrupted, because the interrupted step keeps its index
Goal, Limits	for the run; a zero MaxSteps means DefaultLimits
Reconcile	the consumer's Config.Reconcile; nil leaves the runtime to pause an interrupted side effect for an operator
Operator	answers each pending approval with Approve, Reject, or Leave; nil approves every one
Now, NewID	passed to every driver, so a deterministic clock and id sequence make two runs of one scenario record identically
DB	the database file, for a consumer whose own tables share the runtime's file; empty means a file of the harness's own
LeaseTTL	the lease length for every driver; zero means one second
Loop	since v0.4.0, the consumer's own loop over a Gate, run in place of a Driver and its Agent. See below

The crash points

A Crash is a step index and a Point. Each point is somewhere a dying process could stop.

Point	Stops	What a resume then finds
AfterStepStarted	after step.started: the step exists, the agent has not decided	the step interrupted while deciding, and the loop continues
AfterDecided	after step.decided: the decision is recorded, policy has not seen it	the step interrupted while deciding; the request is never executed
AfterToolStarted	after step.tool_started: the start is recorded and the tool never ran	indistinguishable from AfterToolEffect, so treated the same
AfterToolEffect	after the tool returned, its effect done, and before its outcome is recorded: the step is left executing	an interrupted side effect
AfterToolFinished	after step.tool_finished: the step is complete, the next has not started	a run to continue
AfterResumed	after run.resumed; Step is ignored	on an approved resume, the transaction that also recorded the policy decision and the tool start, so the approved request must not run again; on an interrupted resume, the move back to RUNNING before the loop
AfterInterrupted	after step.interrupted: a resume took the run over and died before reconciling or pausing it	the next resume must still know the step was executing
AfterAllowed	after the policy allowed a step's request and before the tool starts: a gate's caller holding an allowed verdict before Execute, or a Driver between its policy and the tool start	the step interrupted while deciding; nothing runs
AfterAttachAllowed	after Attach returned an allowed verdict for an approved step and before Execute; only a Scenario.Loop reaches it	the approval still approved and the run still waiting, and the next Attach runs the grant once

Every pause is a stop as well, with no Crash needed: the loop returns, the Operator decides, and a fresh driver resumes.

How it crashes a run

A crash is a real one as far as the runtime can tell. The hook at the crash point parks the goroutine executing the run, and the harness closes that goroutine's Store, so the heartbeat stops and the lease is never released, which is exactly what a process that died leaves behind. The harness then opens the file afresh and resumes in a new driver. The runtime refuses that with ErrRunLeased until the dead driver's lease expires, as it would a restarted process, so the harness waits out the expiry the refusal names, once per crash, which is why it sets a short LeaseTTL. A value too short for the machine makes a live loop lose its own lease between heartbeats. The parked goroutine is released when the test ends; its writes fail against the closed store and nothing of its reaches the file.

Why not cancel the context. Cancelling the loop's context would be the easy way to stop it, and it is not a crash. A cancelled context returns through the runtime's own paths, which record what they can and release what they hold: a tool cancelled with the caller's context is recorded as interrupted, on a context that outlives the cancellation, and a call that returns for any reason other than pausing or finishing releases its lease with a compare-and-set so another process need not wait out the TTL (see Writing Tools and Run Lease). A process that died does neither. So parking the goroutine and closing the store leaves the file in the state a kill would, and the resume has to establish for itself what happened. The source states that choice as "a crash is a real one as far as the runtime can tell"; the reasoning above is what the runtime's own documentation says about the paths a cancellation takes.

The invariants it asserts

When the run ends, terminal or left waiting, Run reads it from a fresh open of the file and reports every violation with t.Errorf, so a test sees all of them, and returns the Result regardless.

• A step never executes twice on its own. Every execution past the first is under an approval of that step.
• An approved request executes at most once. No approval is the origin of two tool starts.
• Every recorded tool start is followed by its finish or by the step's interruption, and a step does not start a tool again before the previous start was closed.
• The tool was never reached more often than the record says. A step's tool calls do not exceed its recorded starts.
• Nothing is left in flight. The run ends terminal or waiting, never RUNNING or INTERRUPTED; no step is left deciding or executing; a step awaiting approval is only under a waiting run.
• No lease is left. The run's stored lease owner is empty.
• Every crash was reached. A crash point the run never arrived at is a failure, so a script that drifted from the test is not silently passing.

Over a loop of your own

A consumer whose loop runs on a Gate sets Scenario.Loop instead of Agent. Each process the harness simulates opens a fresh gate on the scenario's tools, policy, reconciliation, clock, ids, and lease, takes a session with Begin, or with Attach after a pause or a crash, hands it to Loop, and closes it when Loop returns. Loop drives the session until Done, and when Current is not nil, Attach left an approved step open for Execute. The same invariants are asserted (TestRun_GateLoopKeepsTheContractAtEveryCrash). Result.Inputs is empty for a Loop, and SameRenders still needs an agent. A loop that ignores the approved step, stops mid-step, or returns an error is reported (TestRun_BrokenGateLoopsAreReported).

The result

Result carries RunID, the Run, Steps, Approvals, and Events; Calls, every invocation a tool received across every driver, in order, each with its step, tool, arguments, and Returned, which is false for the call a crash at AfterToolEffect stopped after (the tool did its work and the runtime never recorded it); Inputs, what the agent was handed at every decision across every driver; Crashes, the ones that fired; and Resumes, the drivers after the first.

The package's example is the case that matters most. A send tool of the remote-mutation class, crashed after its effect, is not run again by the runtime: the run waits on an interrupted_side_effect approval, the operator approves, and the same step runs once more under that approval.

COMPLETED after 2 resumes
call step 0 send {"to":"ops"} returned: false
call step 0 send {"to":"ops"} returned: true
approval interrupted_side_effect approved

See Interruption and Reconciliation for the contract being checked.

The policy conformance table

A policy is a pure function of the request and the view, and CheckPolicy decides a table of requests through the runtime's own validation first.

testkit.CheckPolicy(t, policy, tools, []testkit.PolicyCase{
    {Name: "send asks", Tool: "send", Args: `{"to":"ops"}`, Want: agentrt.RequireApproval, Kind: "remote_mutation"},
    {Name: "wipe denied", Tool: "wipe", Args: `{"to":"all"}`, Want: agentrt.Deny},
    {Name: "no recipient", Tool: "send", Args: `{}`, Want: testkit.Refused},
})

A PolicyCase is a name, a tool and its JSON arguments (empty means {}), the View handed to the policy (its Run.ID, when set, is the request's too), and what is wanted: Want, the outcome, or testkit.Refused when the runtime must refuse the arguments before the policy sees them. Kind, when set, is the approval kind a require_approval must carry, and Reason, Capability, and Presentation, when set, are substrings the decision's reason, capability JSON, and presentation JSON must contain.

Validation comes first. The case goes through agentrt.CompileTool and ToolSchema.Check, the same acceptance the driver applies, so an unknown tool, arguments that fail the schema, or JSON the boundary rejects are Refused and never reach the policy. Refused is the kit's value, never a policy's. A set of tools NewDriver would refuse fails the test with NewDriver's error.

Disagreements are reported as one table, with what the policy answered and why:

testkit: 1 of 4 policy cases disagree:
case               request            want   got   detail
wrong expectation  wipe {"to":"all"}  allow  deny  side effect destructive is deny by policy

Never

testkit.Never(t, policy, tools, class, outcome, fuzz, views...) asserts that a policy never answers outcome for a tool whose side effect is class, over fuzz.Count generated arguments per such tool that pass the runtime's validation, in each of the views given (the empty view when none are). It is the assertion a table of examples cannot make: "a destructive tool is never allowed", over generated requests rather than the ones its author thought of. A class with no tool fails, because the assertion would hold of nothing.

Schema fuzzing

testkit.Fuzz generates tool arguments from a tool's JSON Schema, with the same schema library the runtime validates with. Generation is deterministic for a Seed and bounded in size.

Field	Default	Meaning
Seed	1	selects the sequence
Count	32	how many arguments Valid returns and at most how many Invalid returns
MaxItems	4	bounds a generated array where the schema does not
MaxLength	32	bounds a generated string, in characters
MaxDepth	4	bounds nesting where the schema recurses: below it, optional properties are left out and arrays are as short as allowed

What it generates for. type (one or several), enum, const, object properties with required, additionalProperties, minProperties and maxProperties, array items with minItems, maxItems, uniqueItems and prefixItems, string minLength and maxLength, numeric minimum and maximum (exclusive too), anyOf and oneOf by picking a branch, and $ref within the schema. A schema using pattern, multipleOf, allOf, not, if, patternProperties, propertyNames, dependencies, dependentRequired, dependentSchemas, contains, the unevaluated keywords, or a dynamic or recursive reference is reported as one the kit cannot generate for, by name, rather than guessed at. Every generated value is checked against the compiled schema before it is returned.

Valid(schema) returns arguments that satisfy it. Invalid(schema) returns near misses: valid arguments with one change the schema refuses (a required property missing, an unexpected one present, a wrong type, an enum outsider, a length, bound, or item count just past its limit, a non-object root), each confirmed invalid against the compiled schema.

The boundary cases come first in Invalid, because they are what the runtime refuses before any schema, and a schema that forbids unknown properties must not be what refuses them:

• nesting past the depth cap (the runtime's is 256; the case nests 300);
• a duplicate object key;
• invalid UTF-8;
• an unpaired surrogate escape;
• truncated JSON;
• two values.

The bad bytes are carried by a string property when the schema has one. A schema with nothing to violate yields the boundary cases only.

Fuzz.Check(t, tools...) puts every tool's arguments through the runtime's acceptance. The valid ones must pass it, then reach the tool and return without a panic; the invalid ones must be refused, as the driver refuses them with an invalid_decision. A tool's own error on a valid argument is allowed, since a schema does not say what a tool accepts. Each call is bounded by the tool's Timeout, as the driver bounds it, and a terminal tool is called like any other.

To validate arguments in a test without any of this, use agentrt.CheckArgs directly; see Writing Tools.

Render equivalence

An agent that builds a model request from the recorded steps must build the same bytes from the same steps, or its recordings stop matching and its runs stop being comparable. testkit.Rendered(t, agent, input) returns the model request an agent makes for a step input: it calls Decide with a model caller that records the first request and answers with an empty end_turn reply, so it is the request a recording keys on.

testkit.SameRenders(t, fresh, crashes...) runs the scenario fresh returns twice, with the same crashes, and fails unless the agent fresh returns renders a byte-identical request from what the two runs handed their agents at every decision, those after a pause or a crash included. fresh is called once per run, so state is not shared between them, and a deterministic Now and NewID in the scenario make the two records identical in ids and times, which an agent that renders either depends on. A failure names the decision and the first message that differs. See Rendering Model Context and Models and Replay.

Re-checking a policy

A policy is a pure function of the request and the view only if it reads nothing else. The conformance table tests it on the requests you write. The re-check helpers test it on a run that happened: they hand each policy evaluation the run recorded back to the policy, rebuilt as the policy saw it, with agentrt.Recheck, and fail the test on a difference. See Re-checking a Run for what is recorded and what a re-check does and does not prove.

// the scenario's own policy decides the run again, after it has run
res := testkit.Run(t, testkit.Scenario{Tools: tools, Policy: policy, Agent: agent, Recheck: true})

// a stricter policy: exactly step 2 would have been stopped
rep := testkit.RecheckDiffers(t, store, runID, stricter, 2)

Helper	Fails the test when
Recheck(t, store, runID, policy)	any evaluation differs from the one recorded
RecheckDiffers(t, store, runID, policy, steps...)	the steps whose evaluations differ are not exactly the ones named, by index
RecheckPartial(t, store, runID, policy, steps...)	as RecheckDiffers, but passes evaluations that are not re-checkable, and a run with none re-checked
Scenario.Recheck	set to true, Run re-checks the scenario's run under its own Policy once it has run; a policy that reads outside state fails it

They fail when nothing was re-checked. Recheck, RecheckDiffers, and Scenario.Recheck fail a re-check that proves nothing: one that re-checked no evaluation, a run recorded before v0.5.0, one whose only evaluation met a policy error, one that asked the policy nothing, and any with an evaluation left out as not re-checkable (TestRecheck_FailsWhenNothingWasRechecked). A test that passes because it checked nothing is the failure to avoid. RecheckPartial is for a run the record cannot rebuild whole. Each helper returns the report, and a failure includes it as text.

A policy that decides from its request and view passes, and one that reads a clock, a counter, a file, or a service may fail (TestRecheck_ScenarioPolicyThatDecidesFromItsInputPasses, TestRecheck_PolicyThatReadsOutsideStateFails). The failure is a finding about the policy. A policy that reads outside state for a reason it can name, as repo-steward's reads workspace facts, cannot be re-checked exactly; it keeps Scenario.Recheck off and tests what it can with the conformance table.

What it cannot reach

The harness crashes at the points listed above and at every pause, and no others, because the public API reaches nothing else. It cannot crash inside one transaction, between a tool's own journal writes, or during a model dispatch. A consumer whose tools keep their own journal still injects faults at those writes itself, as Testing describes, and drives Resume; the harness gives it the runtime's side of that bargain.

It also asserts the runtime's contract, not the consumer's: that a tool is safe to run again, or that a reconciliation answers correctly, is the consumer's to test, with the harness doing the crashing.

A reference agent

The roadmap asked this item to examine the generic parts of both consumers' model agents and decide whether a reference Agent belongs in the runtime. The examination found none worth shipping yet: about ten generic lines remain in each consumer's agent. The one generic candidate is casework's raw model-turn table, which belongs in the runtime as a store table, not as an Agent.

Adoption

The roadmap's done-when is that both consumers replace their own interruption and policy tests with the kit. Both did on 2026-10-03: repo-steward's policy tests are conformance tables with a Never assertion and Fuzz.Check over its tools, and casework's crash-recovery and policy contract tests run on Run, CheckPolicy, Never, and Fuzz.Check, with every old assertion kept. See Status and Roadmap.

Next

Measuring Agents is for scoring what an agent does rather than proving what the runtime promises. Writing a Policy and Writing Tools are what this page tests. Re-checking a Run is what the re-check helpers are built on.