Run Lease

Run Lease

A run found RUNNING in the database is in one of two states: a loop somewhere is executing it, or the process that was executing it died. Nothing in the row used to tell the two apart, so Resume assumed the second, and a second process could take over a run whose loop was still alive. The compare-and-set guards stopped the live loop at its next write, but its tool call in flight still completed, and the newcomer could enter the same step through reconciliation while it did. That is a side effect executed twice.

The lease is the smallest record that answers the question. A live owner renews it; a dead one cannot. It arrived in v0.3.0 as migration 5, and ADR 5 records the decision. This page is what it does, read from lease.go, store.go, driver.go, and resume.go.

What it is

Two columns on the run's own row: lease_owner and lease_expires_at. No separate table, because that would be a second place a run's state lives, and no event for the ordinary acquire, renew, and release, so the event sequence of an ordinary run, which consumers' tests and replays compare, is unchanged.

A run that is not RUNNING holds no lease. Pausing or finishing a run releases it in the same transaction that moves the run.

The owner

The stored owner is Config.LeaseOwner followed by / and a random suffix drawn once per Driver. With no LeaseOwner, it is the random part alone.

driver, err := agentrt.NewDriver(agentrt.Config{..., LeaseOwner: "steward@build-3"})
// leases are stored as "steward@build-3/9c41e0b27d5a"

The name is for an operator reading the row. The suffix is what makes it an identity. A configured name alone identified nothing: a sweep reproduced a side effect executed more than once when two replicas, or a restarted process, used the same LeaseOwner, because each took the other's live run as its own. With the suffix, no two drivers ever share an owner, in one process or two, and a restarted process waits out the leases its previous life held like any other process would, because nothing proves the old process is gone.

One driver also refuses a second Start or Resume of a run it is already executing in another call, with ErrRunLeased, rather than taking over the lease it holds itself.

When it is taken

• Start: the transaction that inserts the run as RUNNING stamps this driver's lease.
• Resume of a waiting run: the move from WAITING_FOR_APPROVAL to RUNNING takes the lease, in the transaction that also records the resume-time policy decision and the tool start. Two resumers race on that compare-and-set; one wins and the other gets ErrRunState.
• Resume of a RUNNING run whose lease has expired, or that has none: the lease is taken, and the steps in flight marked interrupted, in one transaction. Taking over a lease another owner let expire, or this driver's own expired one, is recorded as a lease.taken_over event naming the previous owner, its expiry, and the new owner, because that is the moment an operator will ask who ran what.

When it is renewed

By a heartbeat goroutine on the wall clock, every third of Config.LeaseTTL, for as long as the call executing the run lives. It is time-based rather than per step because a tool or model call can outlast any TTL a crash should be detected within: a lease renewed only between steps would expire in the middle of exactly the long calls it exists to protect.

The renewal is one UPDATE outside any transaction, compare-and-set on the owner. When it finds another owner, the lease is lost. A renewal that fails with an error is retried at the next tick; if none succeeds before the expiry passes, the lease is lost by expiry.

When it is released

When the call returns. A call that left the run paused or finished released it in that transaction; any other return (an error, a cancelled context) releases it with a compare-and-set on the owner, so another process need not wait out the TTL. A panicking call releases its lease too (TestLease_HeartbeatStopsOnReturnAndOnPanic).

An operator's Cancel needs no lease. It is the operator's authority, it releases the lease along with the run, and a live loop stops at its next write.

Reading a lease

Store.Lease(ctx, runID) returns the stored lease without taking or changing it: the owner string as stored, which is Config.LeaseOwner, a /, and the driver's random suffix, and the expiry by that driver's wall clock. A run nobody holds, because it is paused, finished, or was released by the call that executed it, has an empty owner and a zero time. A lease whose time has passed is stale, not released: Resume takes such a run over. An unknown run is ErrNotFound. It is for a tool that sits beside a consumer and wants to know who is running what, and testkit uses it to assert that no run is left leased after a crash and resume. It was added in v0.3.1 and writes nothing.

LeaseTTL

Config.LeaseTTL is how long a lease lasts without renewal. Zero means DefaultLeaseTTL, 30 seconds. It is the time a crashed process blocks Resume of its run, so it is the knob for how fast recovery is:

LeaseTTL	A crash blocks Resume for up to	The heartbeat writes every
30 s (default)	30 s	10 s
6 s	6 s	2 s

Lower is faster recovery and more writes; a process suspended or starved for longer than the TTL loses its runs. A fast resume after a crash comes from a lower LeaseTTL, not from a stable LeaseOwner: there is no instant takeover by name.

Lease times are read from the wall clock, never from Config.Now. The number and order of Now readings is a contract with consumers whose deterministic clocks stamp their own records (see Testing), and a deterministic clock that never advances would never expire a lease anyway.

ErrRunLeased and ErrLeaseLost

ErrRunLeased is a struct carrying RunID, Owner, and ExpiresAt, and it matches ErrRunState under errors.Is. Resume returns it, without reading Config.Now and changing nothing, when the run is RUNNING under a live lease of another owner, or when this driver is executing it in another call. A loop's write also finds it when another owner has taken its run.

ErrLeaseLost is what Start or Resume returns when its loop lost the lease: another owner took it, or it expired unrenewed. The loop stops before its next side effect and leaves the step in flight to the next owner.

What a consumer does with ErrRunLeased

It is not a failure of the run. It means the run is being executed elsewhere, or was until a process died and its lease has not yet expired. A consumer that calls Resume on startup to recover runs after a crash should expect it for up to one TTL:

run, err := driver.Resume(ctx, runID)
var leased agentrt.ErrRunLeased
if errors.As(err, &leased) {
    // Someone holds it until leased.ExpiresAt. If that process is alive it
    // will finish or pause the run; if it died, Resume succeeds after then.
    log.Printf("run %s is leased by %s until %s", runID, leased.Owner, leased.ExpiresAt)
    return retryAfter(time.Until(leased.ExpiresAt)) // the consumer's own scheduling
}

Retrying after ExpiresAt is safe, because a live owner will have renewed the lease by then and Resume will say so again; there is no way to wait less. Reporting it to an operator is equally reasonable. What a consumer must not do is treat it as fatal and cancel the run, which would cancel a run another process is executing correctly.

The renewal connection

On a file-backed store the heartbeat renews on a database connection of its own. On the store's single shared connection it waited behind whatever held that connection, and a consumer holding Store.DB() with a transaction or unclosed rows starved it until another process took a live run over (TestLease_HeldConnectionDoesNotStarveTheHeartbeat). A :memory: store has one connection, which is its whole database; no other process can take its runs, so it needs no second one.

The renewal connection runs with the same hardening as every other: trusted_schema off and SQLite's defensive mode on.

What happens to a tool call when the lease is lost

Two writes start a side effect: the one that records step.tool_started, and the one that reserves a model call before dispatch. Both require the lease held and unexpired by the wall clock, not just held. So a loop stalled past its expiry starts nothing, even if nobody has taken the run (TestLease_LapsedLeaseStartsNoSideEffect), and a loop that has lost its lease stops before its next tool call or model request (TestLease_LoopThatLosesItsLeaseStopsBeforeTheNextToolCall, TestLease_NoModelCallAfterTheLeaseIsLost).

A tool call already running cannot be recalled. When it returns, its outcome is recorded only if its loop still holds the lease. Otherwise the step stays as it was, executing, for the new owner, whose Resume marks it interrupted and reconciles it, which is exactly the contract a crash already had: the new owner cannot know whether the call took effect, so it is an interrupted side effect. The dead owner's late outcome is not written (TestLease_ExpiredLeaseIsTakenOverAndReconciledOnce).

The one exception is an operator's cancel. A tool running when an operator cancels the run still has its outcome recorded, as a step.tool_finished event marked late, appended after run.finished, while the run stays cancelled.

The grace lease at migration

A database written before v0.3.0 has runs with no lease, and a run left RUNNING by an older version may still be executing in a process of that version, which knows nothing of leases. So when migration 5 is applied, every RUNNING run is given a lease DefaultLeaseTTL long under the placeholder owner agentrt:pre-lease-version. Resume refuses it until that passes, then takes it over with a lease.taken_over event (TestStore_V021DatabaseMigratesWithItsRuns). After that the run is free.

What leases do not protect against

An older binary. A process or an agentrt of a version before the lease ignores both columns. It will resume a run a current process is executing, and nothing in the file can stop it. Stop every process of an older version before a current one first opens the database; see Upgrading to v0.3.

Clock skew between hosts. Expiry compares wall clocks across processes. Processes sharing a database file must run on one host, or on hosts whose clocks agree to well within LeaseTTL. A host whose clock runs ahead sees live leases as expired.

Network filesystems. SQLite on NFS or SMB is unsupported: file locking there is unreliable enough that two processes can each believe they hold the write lock, and the lease is written through that lock.

A suspended process. A lease is renewed only while its process runs. A process suspended longer than LeaseTTL finds its lease expired when it wakes and starts no further tool or model call, but another process may already have taken the run, and the call it was in the middle of when suspended cannot be recalled.

Cost

Measured in Performance: time per step moves by less than run-to-run noise. A step pays about ten allocations in the guards, one wall-clock read per tool call, and, on a file-backed store, a second connection and a write every TTL/3 per running call.

Next

Interruption and Reconciliation is what a takeover leads to. Persistence and Inspection covers the rest of the store.