Interruption and Reconciliation

Interruption and Reconciliation

The problem

A process dies while a step is executing. The tool may have done nothing, part of its work, or all of it. The runtime cannot know, because side effects live outside it. Re-executing would risk doing the work twice; assuming success would risk claiming work that never happened.

There is a quieter version of the same problem. Before v0.3, a run with no reconciliation marked the step interrupted and went back to the agent, which saw that its last call had no result and, reasonably, asked for it again in a new step. The runtime never re-ran the step itself, but the effect was identical: a payment, a push, or a message sent twice, with nothing in the record to say the second was a repeat of a call whose outcome nobody knew. v0.3 closes that path too.

How a run comes to be interrupted

• The process dies, by a crash, a kill, or power loss. The run is left RUNNING with a step deciding or executing. Its lease stops being renewed and expires, by default within 30 seconds.
• The caller's context is cancelled. A cancellation while the agent is deciding leaves the step in flight and Start or Resume returns the context's error; a tool that fails because of the cancellation is observed as interrupted, because its outcome is unknown. What a tool did, or a model call cost, is recorded even after the cancellation, on a context that outlives it.
• The loop loses its lease to another process, or lets it expire. It stops before its next side effect; a tool already running finishes, but its outcome is not recorded, and the step is left executing for the new owner.

In every case the run stays RUNNING. The runtime does not write the INTERRUPTED run status: a later Resume finds the run by its lease.

What Resume does

On a run found RUNNING:

1. It respects a live lease. If another owner holds the run's lease and it has not expired, Resume returns ErrRunLeased and changes nothing. The run is being executed elsewhere, or a dead process's lease has not yet run out.
2. It takes the run over, in one transaction: it takes the lease, writes lease.taken_over if another owner (or this one) had held it, and marks every step in flight interrupted with an interrupted observation naming the status it was in, writing step.interrupted for each. Of two processes resuming at once, one takes the run and the other gets ErrRunLeased.
3. It refuses to continue from a damaged record. If a step or approval row will not decode, the run fails as internal_error without rewriting the row.
4. It asks the consumer, if there is a Config.Reconcile, and acts on the answer.
5. Otherwise it decides by what was in flight, as below.

With Config.Reconcile

Reconcile receives a RunView (copies of the run, its steps, and its approvals) and returns a Reconciliation:

Outcome	Effect
continue	the loop proceeds with the next decision; the agent sees the interrupted observation
completed	the interrupted operation had already succeeded; the run completes with Result, which must be usable JSON
waiting	the interrupted tool call pauses on the approval in Pause
conflict	external state contradicts the record; the run fails with reconcile_conflict and Detail

An outcome outside these four, an error from Reconcile, or a completed result that is not usable JSON fails the run as internal_error.

waiting needs Pause. Reconciliation.Pause is the require_approval decision to pause on, exactly as a policy would return it, and the interrupted step goes back to awaiting_approval through the ordinary pause, with a new hash-bound approval that Approve and Resume treat like any other. It only works when there is something to pause: a step that was executing a tool call, whose tool is still registered, whose arguments were JSON, and whose arguments still pass the tool's schema. Otherwise the run fails as reconcile_conflict rather than waiting on nothing; a step whose arguments now fail the schema is ended as invalid_decision first. A Pause whose outcome is not require_approval, or whose capability or presentation is not usable JSON, fails the run as internal_error.

Without Config.Reconcile

What was in flight	What happens
a tool that is not ReadOnly was executing	the run pauses for an operator on an interrupted_side_effect approval
a ReadOnly tool was executing	the loop continues; the agent sees the interrupted observation
the agent was deciding	the loop continues
nothing	the loop continues

The pause is ADR 6. The approval's kind is agentrt.InterruptedSideEffect, its request is the step's recorded request, its capability names the tool and arguments, and its presentation carries the tool, the arguments, the side-effect class, and this notice:

This tool call started, and the process stopped before its outcome was recorded, so whether it took effect is unknown. Approving runs it again with these arguments; rejecting ends the run.

Approving and resuming runs the same request again, once, on the same step, after the same schema check and policy re-evaluation as any grant. Rejecting ends the run with the tool not called again. If the tool is no longer registered, or its arguments no longer pass its schema, the step cannot wait for an approval and the run fails as reconcile_conflict. A ReadOnly call is exempt because running it again changes nothing.

The operator now has to decide something the runtime cannot check. That is the point: a consumer that wrote no reconciliation has said nothing about how to find out whether the call took effect, so the question goes to a person who is shown exactly the call that was in flight and told its outcome is unknown. The alternative considered and refused was to document at-least-once delivery and require every tool to be idempotent, which most outside systems a tool talks to cannot be.

A resume that marked the step interrupted and stopped before pausing it, because it crashed too, leaves the step for the next resume, which still finds it as the step whose outcome is unknown.

A tool still running when an operator cancels

An operator's Cancel needs no lease and does not interrupt a tool already running. When that tool returns, the run is cancelled and its step has been failed by Cancel, but the side effect happened, and its record must survive. So the outcome is appended to the cancelled run as a step.tool_finished event marked "late": true, carrying the observation, after run.finished. The run stays cancelled and the step row keeps what Cancel wrote; the event is the record of the side effect.

What the runtime guarantees, exactly

• It never executes a step twice of its own accord. A tool is started only by a write that records step.tool_started under a live lease, and a resume never re-enters a started tool by itself.
• It never lets the agent ask anew for an interrupted side effect without someone deciding: an interrupted call that is not ReadOnly runs again only when the consumer's reconciliation says so, or an operator approves it. An approved re-run is the same step executing again, under an approval that says the first attempt's outcome is unknown.
• It always records the interruption: the step, the observation, the event, and, on a takeover, who held the run before.
• It does not know whether an interrupted call took effect. Nothing in the runtime can; that is what reconciliation, or the operator, is for.

What the consumer does

The consumer journals each side effect before it starts, with enough identity to recognize its result: a hash of the before and after states, a marker in a remote object, a deterministic id. Reconciliation reads the journal and the world and decides. It never re-runs an action because a journal row says it was in progress; it finishes from the journal when the world matches, restores when nothing happened, asks to wait when a person must look, and reports a conflict when neither.

Repo Steward's manifest promotions, proposal commits, and publications are worked examples; see its Reliability and Recovery page.

Testing

Fault points compiled in under a build tag let a consumer build a binary, kill it at a chosen point, run Resume, and assert the outcome; the runtime itself has no fault points. The runtime's own tests cover each path: TestResume_ReconcileOutcomes for every outcome, TestResume_ReconcileWaitingPausesTheInterruptedRequest and TestResume_ReconcilePausesOnlyAValidExecutingStep for waiting, TestResume_InterruptedSideEffectWaitsForAnOperator and TestResume_InterruptedReadOnlyOrDecidingStepContinues for the pause without a reconciliation, TestLease_ExpiredLeaseIsTakenOverAndReconciledOnce for a takeover, and TestCancel_ToolOutcomeIsAppendedLate for the late event.

Next

Run Lease is how Resume tells a dead run from a live one. Approvals and Resume is what happens after the pause.