Persistence and Inspection

Persistence and Inspection

The tables

Table	Rows
runs	id, goal, status, reason, result, limits, totals for steps, model calls, tokens, cost, active time, timestamps, and the lease owner and expiry
steps	id, run, index, status, decision, policy decision, observation with content hash, the hash of the tool's spec, the policy's identity, timestamps
model_calls	id, run, step, attempt, status, model, usage, latency, cost, error
approvals	id, run, step, kind, capability, presentation, request, hash, status, decided by, note, expiry
events	sequence, run, step, time, type, payload, hash
tool_specs	each distinct tool spec, once, under the hash of its canonical JSON
schema_migrations	applied versions

Tables are the state. Events are the explanation. Nothing is reconstructed by replaying events.

The file is the operator's

Anyone who can write the database can forge an approval, because an approval's hash has no secret in it. So the file is kept owner-only, by both openers:

• A new database is created 0600, under any umask, and SQLite gives its WAL files the same mode.
• A database or WAL file group or others can write is refused with agentrt.ErrInsecureMode, which names the file, its mode, the reason, and the fix (chmod 600 and chown to the user that runs the runtime). So is a WAL file owned by another user than the database, and a WAL path that is not a regular file.
• A database others can only read is tightened, WAL files included, by a writer that owns it: OpenStore, or OpenExisting opened for writing. A read-only open changes nothing.
• On Windows these rules are skipped, because file modes there do not mean owner, group, and others.

A database may also come from someone else, as one an operator is asked to inspect:

• A trigger or a view is refused by both openers with agentrt.ErrUnsafeSchema. The runtime creates neither, and either would run the file author's SQL inside the runtime's own statements.
• Every connection runs with trusted_schema off and SQLite's defensive mode on, the lease heartbeat's included.

Opening

OpenStore(path) opens or creates a database and applies migrations. It follows a symbolic link, because the consumer's own configuration names its database. ":memory:" opens an in-memory store for tests.

OpenExisting(path, readOnly) opens a database without creating or migrating it, which is what an operator's tool wants: a mistyped path is an error, not a new empty database, and a newer binary does not change the schema under the consumer that owns the file. With readOnly it never writes to the file and never changes its mode; it can read a WAL database while a writer is live, and one in a directory it cannot write. It opens the file named: a symbolic link is refused with agentrt.ErrSymlink, and anything else that is not a regular file is refused too. A database whose schema version is not this build's is refused with ErrSchemaVersion, whose message gives both versions. cmd/agentrt uses it.

A path containing ?, #, or % opens the file it names.

Migrations

Migrations are forward only, and a released one is never edited (TestStore_ReleasedMigrationsAreUnchanged). They are read and applied in one immediate transaction, so two processes opening the same new file at once are safe: the second waits and then finds the work done. A database a newer build migrated is refused by OpenStore with ErrSchemaVersion rather than written by code that does not know its schema.

Migration 5, in v0.3.0, is the run lease: two columns on runs. It is the first since v0.1, and a v0.2.1 database migrates with its runs intact. When it is applied, every RUNNING run gets a lease one DefaultLeaseTTL long under the placeholder owner agentrt:pre-lease-version, because a process of the older version may still be executing it. Every process and operator binary of an older version must be stopped before v0.3.0 first opens a database; see Upgrading to v0.3.

Migration 6, in v0.5.0, adds tool_specs, steps.spec_hash and steps.policy_id, and events.hash, and computes the hash of every event already stored, in seq order, in the migration's transaction, which takes time in proportion to the log. A v0.4.0 database migrates with its runs intact and its events chained, and a v0.4.0 binary refuses it afterwards with ErrSchemaVersion. Every process and operator binary of v0.4.0 or earlier must be stopped before v0.5.0 first opens a database, because one still writing appends events with no hash; see Upgrading to v0.5. The chain is explained in Re-checking a Run.

Durability and concurrency

Every commit is synced. A store opened for writing uses synchronous=FULL. In WAL mode NORMAL can lose the last commits on power loss, and the last commit may be the step.tool_started of a tool that ran, which the next owner would then treat as never started. That is one fsync per transaction; see Performance for what it costs.

Every transaction begins IMMEDIATE and waits on a five-second busy timeout. It takes the write lock up front, so a read-then-write transaction, including one a consumer runs on DB(), waits for another process's writer instead of failing when it tries to upgrade.

One connection, plus one for the lease. The store keeps a single SQLite connection: SQLite has one writer, and an in-memory database is per connection. A file-backed store opened for writing also keeps a second connection used only by the lease heartbeat, so a consumer holding DB() with a transaction or open rows cannot starve the renewal. Holding the shared connection still stalls every write the loop makes, and a loop stalled past its lease's expiry forfeits the run.

How a transition is written

After the v0.3 audit, the write structure is one transaction per state transition, with its events:

• A step is four transactions: started, decided, the policy decision with the tool start, and done.
• A step and the run it ends commit together: complete, fail, a terminal tool, a tool abort, a policy abort, a limit, loop detection. A crash cannot leave a finished terminal step under a running run.
• A resume's move to RUNNING, its resume-time policy decision, and the tool start it approved commit together, so a crash between them cannot drop the approved request.
• Run and step status transitions are compare-and-set inside the transaction that makes them, and a loop's writes to a RUNNING run also require its lease. Of two writers, one wins and the other gets ErrRunState (or ErrRunLeased); an operator's cancel is never overwritten.
• The transaction helper rolls back on panic, so a panic inside it never leaves the single connection inside an open transaction.

Every write goes through Store.tx, which opens a transaction, runs the mutation, appends the events, commits, and only then delivers the events to the observer. An observer never sees an event whose state did not commit.

Consumers' tables

A consumer may keep its own tables in the same file under its own migrations, and may write them in the same transaction as the runtime's when it has a reason to. Store.DB() exposes the connection for that purpose, with the caveats above: a transaction on it must not call Store methods or the driver, and holds up the loop until it ends.

Reading

Full reads, for code that owns the database: GetRun, ListRuns, ListSteps, ListModelCalls, ListApprovals, GetApproval, ListEvents, and PendingApprovalIDs, which maps every waiting run to its pending approval ids in one query.

Lists follow insertion order, not timestamps: ListRuns is newest first by insertion, and steps, approvals, and events are in the order they were written. Stored timestamps drop trailing zeros and do not sort as text, so ordering by them was wrong, and the choice of which approval is a step's latest follows insertion order too.

A row that will not decode does not make its run unreadable. ListSteps and ListApprovals return it with DecodeError set and the undecodable fields empty. The run can be listed, shown, and cancelled; a driver asked to continue it fails it as internal_error without rewriting the row; such an approval cannot be decided.

Page reads, for a front end reading a database it may not trust: ListRunsPage, GetRunCapped, ListStepsPage, ListApprovalsPage, ListEventsPage, and PendingApprovalIDsOf. Each reads at most the rows asked for, after an offset; caps every text column at MaxPageText (64 Ki characters), keeping the head and a marker giving the full length, or for a JSON column replacing it with {"truncated":true,"length":n,"head":...}, which is still JSON; reads an oversized step or approval column as its identifying fields; and takes its total from a COUNT, so a front end can say how much it did not show without reading it. Neither many rows nor one huge row can exhaust its memory. cmd/agentrt's runs, show, and events read only through them; approve, reject, and cancel read the one run and approval they decide whole, because a decision is bound to the approval's hash.

Reads for tools that sit beside a consumer. Three reads, added in v0.3.1, exist for programs that follow or present a database rather than own it:

• Store.Lease(ctx, runID) returns a run's stored lease: the owner of the driver that holds it and when it expires by that driver's wall clock. A run nobody holds, whether paused, finished, or released by the call that executed it, has an empty owner and a zero time. A lease whose time has passed is stale, not released: Resume takes such a run over. ErrNotFound when there is no such run. See Run Lease.
• Store.ListEventsAfter(ctx, runID, seq, limit) returns at most limit events with a Seq greater than seq, in Seq order: of one run when runID is set, of every run interleaved in commit order when it is empty. Each text column is capped at MaxPageText, and a time that does not parse reads as zero, as in ListEventsPage. Seq is assigned in the transaction that commits the event, so it rises in commit order across the database and is never reused, and the last Seq returned is a cursor a follower resumes after. export.Follower follows the table through it and never writes; see Exporting Events.
• Store.ApprovalReason(ctx, runID, approvalID) returns the reason of the policy decision that paused the run on an approval, capped at MaxPageText characters, read from the approval's approval.requested event, because the approval row has no reason of its own. While the approval is pending the step's recorded policy agrees; once it is granted and resumed the step holds the policy's decision at resume, and this still returns the one that asked. ErrNotFound when the run has no such approval; empty when the approval has no approval.requested event. approver.Pending.Reason is this value, bounded for showing. See Approvals and Resume.

The view package is the read model above them, so a consumer's inspection command does not have to invent one: RunSummary and StepSummary are a run and a step as an operator needs to see them; Runs, Summary, Steps, and Detail build them from full reads, and RunsPage and DetailPage from page reads; PendingApproval implements the rule for choosing the approval an operator means. The model totals come from the run row, so a summary costs no extra query.

cmd/agentrt is a thin renderer over view and is what the runtime ships instead of a UI: runs, show, events, approve, reject, and cancel, in aligned text or -json, against -db or AGENTRT_DB. It has no resume, because resuming continues the loop and that needs the consumer's agent, tools, and policy. See The Operator Command.

Observing

Config.Observer receives every event after its transaction commits. Use it for live tracing; it is not a substitute for the log, which is durable and ordered by sequence.

The trace package has the two observers worth having: trace.Writer(w) prints one aligned line per event with a date, time, and zone, and trace.JSONL(w) writes one JSON object per event with HTML escaping off, for a collector. WriterWithErr and JSONLWithErr also report the first write error. All are safe for concurrent events, because the runtime delivers them from whichever goroutine committed them. events replays a stored log through the same two, so what an operator reads afterwards is what they watched.

To watch the log from outside the process that owns the database, in a tool the operator already has, follow it with export.Follower and send it to JSON Lines or, through the export/otel module, to OpenTelemetry spans: see Exporting Events.

Event types

Run lifecycle: run.created, run.started, run.finished, run.resumed, and lease.taken_over when a resume takes over a run whose lease another owner, or this one, let expire. Steps: step.started, step.decided, step.policy, step.tool_started, step.tool_finished (marked late when it follows an operator's cancel), step.failed, step.interrupted. Approvals: approval.requested, approval.decided. Limits: limit.exceeded, loop.detected. Models: model.dispatched, model.completed, model.failed.

Acquiring, renewing, and releasing a lease write no events, so an ordinary run's event sequence is the same as before leases existed. EventRunInterrupted (run.interrupted), which the runtime defined but never wrote, was removed; an interruption is recorded per step.

What the records are for

Answering, after the fact, what the agent asked for, what the policy said, what ran, what it produced, what it cost, who approved what, and who held the run when. The decision is recorded before validation, so an invalid request is in the log, with arguments that were not JSON kept verbatim in Decision.InvalidArgs. The model's stated reason is recorded as data. Every model attempt, including retries and ambiguous ones, has a row.

Trust

Within the rules above, the database is trusted as the operator's filesystem is. The mode checks and the trigger and view refusal stop another account from writing the file, or a crafted file from running SQL through the runtime's own connection; they do not make the database a trust boundary against its owner. Hashes on observations and approvals catch corruption and code paths that mutate a record after it was fixed, not someone who can write the file: root, or the owner's own account. SQLite on a network filesystem is not supported. See Security Model.