Re checking a Run

Re-checking a Run

An audit log says what was decided. Before v0.5.0 nothing in it showed that the same policy would decide the same again, and nothing showed that the log itself was the one the runtime wrote. v0.5.0 adds three things that answer those two questions, as far as they can be answered:

• each decision records what it can be checked against: the tool's spec by hash, the policy's identity, and what the policy saw of the run;
• the events form a hash chain, which agentrt verify checks;
• a finished run can be replayed through a policy with agentrt.Recheck, which reports where the policy decides differently.

This is roadmap item 10, and decision 9 has the reasoning. The limits of each are part of the design and are stated on this page, not in a footnote. Upgrading to v0.5 says what an upgrade needs.

What is recorded

A policy decides from three things: the request, the spec of the tool, and the run as it stood. The request is already in the step. The other two were not recorded, so v0.5.0 records them.

The tool's spec, by hash. A tool_call step naming a registered tool records Step.SpecHash with its decision: the hex SHA-256 of the canonical JSON of the whole ToolSpec, Timeout and Terminal included, because the policy is handed all of it. Each distinct spec is stored once in the new tool_specs table and read back whole with Store.ToolSpec, which refuses a spec whose stored JSON no longer hashes. A tool registered again with another description, timeout, or terminal flag is another spec, and both stay readable (TestToolSpec_StoredOnceAndReferencedByEachStep, TestToolSpec_ChangedSpecIsAnotherHash). run.created gains tools: the hash of every registered tool's spec, in name order.

The policy's identity. A policy with a PolicyID() string method implements IdentifiedPolicy. Its identity is recorded on each step it evaluated (Step.PolicyID) and in that step's step.policy event. NewDriver and NewGate read it once and refuse one longer than 256 bytes, not valid UTF-8, or holding a control character (TestPolicyID_RecordedOnStepAndEvent, TestPolicyID_MalformedIsRefused). A policy without the method records an empty identity. What goes into the identity is the policy author's choice, and it should change whenever a decision could: a version string, or a hash of the rules. SideEffectPolicy is a map and cannot name itself, so it does not implement the interface; examples/recheck wraps one to give it a name.

What the policy saw. The step.policy event gains policy_id (omitted when empty), spec_hash, the spec the policy was handed, and view: the run's status, step count, model calls, input, output, and cached tokens, estimated cost, and active time as the RunView carried them, and how many steps and approvals it held. Those counts are the run's first ones. With run.created, the steps as stored, and the approvals as they stood, that rebuilds the run the policy was handed. In the loop the run is the one read when the step started, so a model call the step's own decision made is not in it; on resume it is the waiting run. A step.policy written by a reconciliation's or an interrupted side effect's pause has no view or policy_id, because no policy decided it. Recording all of this reads no clock (TestDriver_ClockReadingsAreStable), and the tables stay the state and the events stay the explanation: nothing is replayed to reconstruct a run.

The event chain

Every event's hash is the hex SHA-256 of seven netstrings (the length in bytes, a colon, the bytes, a comma): the previous event's hash, empty for the first; the event's seq in decimal; and its run id, step id, time, type, and payload exactly as stored. It is computed in the transaction that writes the event, which already holds the write lock and reads the head, so concurrent writers, in one process or several, extend one chain (TestVerifyEvents_ConcurrentWritersKeepTheChain). Another language can recompute it with nothing but SHA-256; the encoding is pinned by a literal (TestEventHash_Golden).

The chain runs by seq across every run, in commit order, and not per run. An event deleted from the middle of another run's record, or a run deleted whole, breaks it where a per-run chain would not. Event has no hash field, so a trace is byte for byte what it was.

Store.VerifyEvents(ctx, from, to) walks a range of the chain a page at a time, reading a long field in chunks so no event is held whole, and reports the first event that does not chain in a VerifyReport with a ChainBreak. A break is an event whose stored hash is not the one its fields and its predecessor's hash give, or whose seq does not follow its predecessor's: an altered payload or time at its seq, a deleted event at the next seq, two swapped events at the first (TestVerifyEvents_FindsTheFirstAlteredEvent, TestVerifyEvents_ReadsLongFieldsInChunks). Store.ChainHead and Store.EventHash read the head and any event's hash.

agentrt verify

agentrt -db runs.db verify [-from N] [-to N] [-head SEQ:HASH]

It prints the chain's head, then "intact" with the seq and hash of the last event checked, or the first break with the hash expected and the hash found:

head: seq 20 27e4a4f4...
intact: 20 event(s), seq 1 to 20
last checked: seq 20 27e4a4f4...

head: seq 20 27e4a4f4...
broken at seq 15: the stored hash is not the one the event's fields and the previous event's hash give
  expected 5e5b76ea...
  found    236bb02f...
error: the event chain is broken at seq 15

(Hashes shortened here; the command prints all 64 hex characters.) It exits 0 when intact, 1 on a break, and 2 on bad arguments before opening anything (TestVerify_IntactChainExitsZero, TestVerify_BrokenChainExitsOne, TestVerify_BadArgumentsAreUsage). It opens the database read-only and changes nothing. -json prints the same as JSON. See The Operator Command.

What the chain proves

Whoever can write the database can recompute every hash after a row they changed, and can cut events from the end, and the chain still verifies. So:

• Without a head kept elsewhere, the chain catches corruption and a careless or partial edit, nothing more. It is not a defence against whoever can write the file. Security Model says who that is.
• Against a head kept where that writer cannot reach, it shows the record unaltered up to that head. -head SEQ:HASH takes a head kept from an earlier check, or from export.Follower.Head, and fails unless the walk from the first event reaches that seq with no break and the event there still has that hash. A chain cut at its end verifies on its own; only a kept head shows it is short. A database cut before the kept seq fails -head, -to at that seq fails, and a kept head whose event has another hash fails (TestVerify_CutDatabaseFailsAKeptHead).
• Events stored before migration 6 carry no claim beyond the backfill. The migration hashed them when it ran. Their hashes say what they held then, not that they were never changed before.
• The state tables are not chained. The chain covers events only. An approval's own hash still binds the approval, as before.
• The chain is not a signature. A secret in the writer's process proves nothing against the writer, and a key kept out of its reach is a deployment the project does not have. Signed events are deferred in decision 9.

Keeping a head

The anchor is a seq and a hash. Keep it somewhere the database's writers cannot write: another host, a log service, a ticket. agentrt verify prints one as "last checked", and the exporter hands one over:

f := &export.Follower{Store: store, Cursor: export.FileCursor{Path: "export.cursor"}}
// ... f.Follow(ctx, sink) ...
seq, hash, err := f.Head(ctx) // the seq delivered up to, and that event's stored hash

Later, agentrt -db runs.db verify -head 20:<hash> checks the database against it. See Exporting Events.

Re-checking a run

rep, err := agentrt.Recheck(ctx, store, runID, policy, agentrt.RecheckOptions{})

Recheck hands each policy evaluation the run recorded back to a policy, rebuilt from the record as the policy saw it, and reports for each whether the answer is the same. An evaluation is a step.policy event carrying a view, so a step that was approved and resumed has two, the second marked OnResume. It writes nothing, reads the events and steps a page at a time, and reads no clock.

What is rebuilt.

• The request. The run and step ids, the arguments as the step's decision recorded them, and the spec the event's spec_hash names. With RecheckOptions.Tools the current spec of each named tool is handed instead, and the arguments are checked against its schema first, so the question becomes whether today's policy with today's tools would have decided the same. Arguments and schema reach the policy in PolicyJSON form, as they do in the loop, so a policy that reads bytes answers alike (TestRecheck_ByteReadingPolicyDecidesAlike, TestRecheck_PrettySchemaHandedAlike).
• The run. Its id, goal, limits, and creation time from run.created, and the rest from the event's view.
• The steps. The run's first ones, as many as the view counted, as stored now. A step before the one evaluated was finished before the policy was asked and is never written again.
• The approvals. The run's first ones, as many as the view counted, each read whole because a decision is bound to all of it, with the status each had when the event was written. That is the status of its last approval.decided before the event, by seq, or pending when there was none. Seq decides and not time, because an operator's clock and a consumer's need not agree. At most 1024 approvals are read.

(TestRecheck_RebuildsTheViewOfTheLoop, TestRecheck_RebuildsTheViewOnResume, TestRecheck_RebuildsTheViewThroughAGate.)

What is compared. The answer is compared with the recorded decision on its outcome and, for require_approval, on the approval kind and the hash of the approval it asks for, computed by the runtime's own function over the recorded request on one side and the re-checked request on the other. A changed capability, presentation, or current spec therefore differs although the outcome does not (TestRecheck_ChangedPresentationDiffersByHash). The reason is reported and is never a difference. A policy error is a difference, and so is a decision that is not usable JSON, unless the recorded decision was refused as invalid for the same reason with the same outcome and kind, which is the same (TestRecheck_InvalidDecisionRefusedAlikeIsSame).

The report

RecheckReport lists each evaluation with its step, seq, tool, the recorded and re-checked decisions, the spec source (SpecRecorded or SpecCurrent), and the recorded PolicyID. Each evaluation's Result is one of:

Result	Meaning
same	the policy answered alike
different	it answered otherwise, or failed to answer
not_a_policy_evaluation	a step.policy that no policy wrote: a reconciliation's or an interrupted side effect's pause (TestRecheck_InterruptedSideEffectPauseIsNotAPolicyEvaluation)
not_recheckable	the record cannot rebuild it; see below

The report also compares the identity recorded with the policy's own: IdentityUnknown when either side has none, IdentityChanged when both do and differ, IdentityUnchanged when every evaluation recorded the identity the policy has now. Three methods answer the questions a gate needs:

• Same() says no evaluation re-checked differs. A report that re-checked nothing is also Same, which is why it is not enough alone.
• Rechecked() counts the evaluations re-checked, the same and the different.
• Complete() says none was left out as not re-checkable.

A gate that means "the policy still decides this run as it did" wants all three. MarshalJSON adds same, differences, rechecked, not_recheckable, and complete. trace.WriteRecheck renders the report as text, one sanitized line per evaluation, differences first; the text and JSON forms are pinned (TestRecheck_RenderingsArePinned, TestWriteRecheck_OneSanitizedLinePerEvaluationDifferencesFirst, ExampleRecheck). A run that is not yet terminal is re-checked up to its current state, with a note. A missing run is ErrNotFound (TestRecheck_PolicyErrorAndRunNotTerminal).

When a run cannot be re-checked

What cannot be rebuilt is reported and not guessed.

• Recorded before v0.5.0. Such an evaluation recorded neither the spec nor the view, and a run created then says so. An evaluation made after the migration, as the resume of a run left waiting, is re-checked (TestRecheck_PreMigrationRunIsNotRecheckable).
• A policy error. A policy error the original run met wrote no step.policy. It is listed at the seq of its step.failed as not re-checkable, "policy failed; no evaluation recorded", and the report is not Complete (TestRecheck_PolicyErrorIsListed).
• A record cut by the page reads. A step.policy event or a step longer than a page (MaxPageText, 64 KiB per field) is not re-checkable, because a policy never saw a cut value. A step.policy carries the decision's capability and presentation, so an approval that asks for one larger than a page leaves the evaluation that asked for it not re-checkable. The evaluations after it, which see that approval whole, are re-checked (TestRecheck_ApprovalLongerThanAPageIsReadWhole).
• An allowed verdict never executed. A gate's Allowed verdict that was never executed wrote no evaluation, because the step.policy of an allowed request commits with its tool start. There is nothing to list.

What a re-check proves

A re-check proves the policy's answer, and only that.

• It does not run a tool, so it says nothing about what a tool did.
• It trusts what it reads. It does not show the record unaltered. That is the chain's work, against a kept head.
• Under the policy a run ran with it is a determinism check. A policy that decides from what it is handed alone answers the same again. One that reads a clock, a counter, a file, or a service may not, and a difference then says that the policy, not the record, changed.
• A policy that reads outside state cannot be re-checked exactly. repo-steward's policy is one: it decides from workspace facts, so its recorded runs would differ when re-checked. That is a finding about the consumer's determinism, which is what the check is there to make, and not a defect of the check.
• Under another policy it names what would change. RecheckDifferent lists the evaluations a stricter policy would have stopped, and what each would have become (TestRecheck_StricterPolicyNamesTheStepsThatDiffer). Recorded specs against current ones is a separate question: a tool whose description changed re-checks the same against what each run recorded, and today's spec makes the earlier run's approvals other approvals (TestRecheck_CurrentSpecsAgainstRecordedSpecs).

In a test

testkit has helpers that run agentrt.Recheck and fail the test:

Helper	Fails when
testkit.Recheck(t, store, runID, policy)	any evaluation differs
testkit.RecheckDiffers(t, store, runID, policy, steps...)	the steps whose evaluations differ are not exactly those named, by index; a step evaluated before and after an approval is named once
testkit.RecheckPartial(t, store, runID, policy, steps...)	the same as RecheckDiffers, but passes evaluations that are not re-checkable, and a run with none re-checked
testkit.Scenario.Recheck	set on a scenario, Run re-checks the run under the scenario's own policy once it has run, which a policy reading outside state fails

Recheck, RecheckDiffers, and Scenario.Recheck also fail a re-check that proves nothing: a run recorded before v0.5.0, one whose only evaluation met a policy error, one that asked the policy nothing, and any run with an evaluation left out as not re-checkable. A check that passes by checking nothing is the failure a test is most likely to hide, so it is refused (TestRecheck_FailsWhenNothingWasRechecked). RecheckPartial is for a run the record cannot rebuild whole. The report is returned in every case. The helpers' own behaviour is pinned by TestRecheck_ScenarioPolicyThatDecidesFromItsInputPasses, TestRecheck_PolicyThatReadsOutsideStateFails, and TestRecheckDiffers_NamesTheStepsASecondPolicyChanges. See Testing a Consumer.

For the proxy

agentrt-proxy recheck -config proxy.json [-run ID | -session NAME] [-limit N] [-current-tools] [-json]

recheck builds the policy the configuration gives the proxy now and hands it every decision the proxy's policy recorded for a call. It re-checks one run with -run, or the session's newest runs (the configuration's session unless -session names another), at most -limit, 50 by default, read from the proxy's index of its calls. It opens the database read-only. Without -current-tools each request carries the spec it recorded. With it, the proxy loads the pinned tools as serving does and each request gets its tool's spec as loaded now, so a description reworded since makes each approval asked for that tool another approval (TestRecheck_CurrentToolsAfterADescriptionChange).

It exits 0 when every run is the same and at least one decision was re-checked. It exits 1 when any differs, when no run was found (a mistyped -session finds none), when no decision could be re-checked, or on an error, and 2 on a usage error (TestCLI_Recheck, TestCLI_RecheckOfNothingFails).

What it covers and what it does not. It covers the gate's policy: the policy map, each rule's outcome, its fixed values, and its rename. The proxy's rules applied before its policy are not re-checked: the duplicate rule, the repeat window, max_pending, the refusal of a rejected or expired approval, and any call refused before the gate. Without -current-tools, no change to a tool's schema or denied parameters is covered either. The summary says so. A re-check of the proxy's recorded calls under the configuration they ran with comes out the same, and under a policy that now denies payments it names the payment's two decisions and nothing else (TestRecheck_ChangedPolicyNamesTheCallsNowDenied).

The proxy's policy now has an identity, Config.PolicyID: agentrt-proxy/sha256: and the hash of the canonical JSON of the policy map and each rule's side effect, outcome, fixed values, and rename, by server (TestConfig_PolicyIDCoversWhatThePolicyDecidesFrom). Nothing else enters it, so changing a timeout or a description keeps the identity. See MCP Proxy.

The example

go run ./examples/recheck

A scripted agent reads, writes, and publishes under a lenient policy that allows all three. The run is then re-checked under a stricter one, which names the step it would have stopped. Last, one event in a copy of the database is altered and the check agentrt verify runs finds it. It needs no model and makes no network call. The output has this shape:

run <id> ran under "rules v1: mutations allowed" and is COMPLETED

re-checked under "rules v2: nothing leaves the machine":
run <id> COMPLETED: 1 of 3 re-checked evaluations differ; policy changed: recorded "rules v1: ...", given "rules v2: ..."; recorded specs
DIFFERENT step 2 seq 15 publish: allow -> deny: side effect remote_mutation is deny by policy
same      step 0 seq 5 read: allow
same      step 1 seq 10 write: allow
would have been stopped: step 2, publish

altered event 15 in a copy; the hash chain, as agentrt verify checks it:
  original: head seq 20 ...; intact: 20 event(s)
  copy: head seq 20 ...; broken at seq 15: the stored hash is not the one the event's fields and the previous event's hash give

TestRun_StricterPolicyStopsThePublishAndVerifyFindsTheAlteredEvent runs it.

Next

Upgrading to v0.5 is what changes for an existing consumer. Writing a Policy is the policy being checked, and Security Model says what the chain does and does not defend. Persistence and Inspection has the tables and the events.