Reliability and Recovery

Reliability and Recovery

The tool is built to be killed at any point and continued. This page lists what happens at each point.

Principles

• Persist before acting. Every side effect has a row written before it is dispatched, with enough identity to recognize its result later.
• Never re-execute blindly. Recovery compares the journal with the workspace or the remote and finishes, restores, or stops. It never repeats an action because a row says it was in progress.
• Stop on contradiction. A state the journal cannot explain is a conflict, reported with the evidence, not a guess.
• Same code path. resume rebuilds the session from persisted options and facts under the same configuration hash and hands control back to the runtime; there is no separate recovery mode.

Interruption points

Killed during	On resume
cloning or profiling	the run directory is incomplete; the run cannot be resumed and a new one is started
baseline validation	the step is marked interrupted; validation records from incomplete steps are never accepted; the baseline runs again
a read tool	the step is marked interrupted with an observation; the agent decides again
apply_upgrade before promotion	the staged files are discarded; the promotion is aborted; selection reopens
promotion, between the two manifest writes	finished from the retained staging copy if it matches the journal, otherwise a conflict
write_file	atomic rename means the file is either old or new; the step is interrupted; the agent decides again
run_validation	the record is not accepted; the agent validates again
freezing a proposal, between commit and ref	the recipe is re-run, producing the same commit id, and the ref is set
fetching the upgrade with the proxy unreachable	not an interruption: the run ends acquisition_failed (exit 6) with the toolchain's message; a later run may succeed
waiting for approval	nothing to recover; approve or reject, then resume
the push	the remote is read: absent branch is pushed again, branch at the commit is recorded, other commit is a conflict
the pull request	found by its marker in any state and recorded; a foreign pull request is a conflict

Runtime-level recovery

The runtime marks an in-flight step interrupted, calls Repo Steward's reconciliation, and acts on its answer: continue the loop, complete the run with a recovered result, wait for an approval, or fail as a conflict. When reconciliation shows publication already succeeded, the run completes as proposal_published without asking the agent anything.

Locks

An exclusive OS lock per run and per data directory. Held by a suspended process, released by a killed one, never taken over. A second process on the same run is refused immediately.

agent-runtime additionally leases a RUNNING run in the database to the process executing it, renewed on a heartbeat. Unlike the OS lock, a crash does not release this lease early: it stays recorded as live until it expires (30 seconds by default), so resume right after a crash is refused with the lease's owner and expiry rather than racing a process that may still be alive. Waiting out the expiry, then resuming again, is the recovery path; see Runs, Approvals and Resume.

Baseline mode and vulnerable selection

Baseline mode is never resumed, so nothing about -select vulnerable is persisted for resume; the selection is part of the run's configuration hash. Scanner and database directories are named by content and never deleted and recreated within a run; a scanner directory that fails verification is left for the operator to remove, and nothing is rebuilt automatically.

Verification

Fault points at each journaled write are compiled in under a build tag. Tests build that binary, kill it at each point, run recovery, and assert the workspace, the journal, and the remote agree. Publication interruption is tested against a fake GitHub API backed by a real bare repository.

Operational advice

• Keep the data directory on a local filesystem the engine can see, owned by you and not writable by other accounts; Repo Steward refuses one that is.
• After a crash, run runs show <id> before resume to see the last step and any pending approval.
• Do not delete a run directory while a process might hold its lock.
• If resume reports a conflict, read the evidence in the result; the remote or the workspace was changed by something other than this tool.
• If resume reports the run is still leased, wait for the expiry it names (30 seconds by default after a crash, under the runtime's default lease TTL) and resume again; do not kill or resume a process that is genuinely still running.