docs(adr): Agent Lifecycle State Machine

docs/adr/agent-lifecycle.md

@@ -0,0 +1,198 @@
+# ADR: Agent Lifecycle State Machine
+
+- **Status:** Accepted
+- **Date:** 2026-08-08
+- **Author:** @brettchien
+- **Reviewers:** Mira (ECS), Jellyfish (control-plane), Falcon (MCP) — all LGTM
+- **Tracking issues:** implementation openabdev/studio#2
+
+> **Y-statement.** In the context of running agents across heterogeneous
+> runtimes, facing the need for one glanceable, runtime-independent notion of
+> "what state is this agent in", we decided a canonical **6-state** lifecycle
+> discriminated by `(desiredStatus, accepting_work, health, identity_verified)`,
+> to get a
+> **single-field dispatch predicate** and a clean native→canonical projection,
+> accepting a sixth state and a per-runtime projection/conformance burden.
+
+---
+
+## 1. Context & Problem
+
+openab runs agents across different runtimes (ECS today; k8s / GKE /
+docker-compose planned). We need one runtime-independent way to say "what state
+is this agent in" that: any engineer reads at a glance; is identical regardless
+of the runtime underneath; and is what the control plane observes and the
+director acts on.
+
+Humans direct; agents do the control. The control plane must classify every
+agent, at any moment, into **exactly one** state.
+
+## 2. Decision Drivers
+
+- **One-glance comprehension** — a small, mutually-exclusive, exhaustive set.
+- **Single-field dispatch** — "may this agent take new work?" should be one
+  field, not a conjunction every caller must remember.
+- **Runtime-independent, decidable projection** — each driver must map native
+  signals onto the canonical set *without ambiguity*.
+- **Honest about faults vs intent vs teardown** — health, admission policy, and
+  terminate-intent are different axes and must not be conflated.
+
+## 3. Decision
+
+Every agent is in exactly one of **6 states**, discriminated by four observable
+axes — `desiredStatus` (running / stopped), `accepting_work` (bool), `health`
+(in-sync & authorized / not), and `identity_verified` (a **latching** bit: set
+true the first time the agent reaches Running, never cleared). The latch is what
+separates `Starting` (never verified) from `Unhealthy` (was verified, now
+faulted) — without it their `(desiredStatus, accepting_work, health)` tuples
+collide. It is CP-observable per runtime: ECS `lastStatus` ever reached RUNNING /
+k8s ever Ready / compose ever healthy.
+
+```mermaid
+stateDiagram-v2
+    [*] --> Starting
+    Starting  --> Running   : identity verified & config live
+    Starting  --> Stopped   : failed to start / cancelled / reclaimed
+    Running   --> Paused    : cordon (director hold)
+    Paused    --> Running   : resume
+    Running   --> Unhealthy : liveness / authz lost
+    Paused    --> Unhealthy : liveness / authz lost
+    Unhealthy --> Running   : recovered (re-proves identity)
+    Unhealthy --> Stopping  : give up (graceful)
+    Unhealthy --> Stopped   : hard loss (OOM / crash / node death), no flush
+    Running   --> Stopping  : stop / replace (desired=stopped)
+    Paused    --> Stopping  : stop / replace
+    Stopping  --> Stopped   : state saved
+    Running   --> Stopped   : reclaim (hard loss)
+    Paused    --> Stopped   : reclaim (hard loss)
+    Stopped   --> [*]
+```
+
+| State | Discriminator | Definition | The one thing that matters |
+|---|---|---|---|
+| **Starting** | desired=running ∧ ¬identity_verified | CP provisions an authenticated config and injects it; the agent proves identity before it runs. | Identity is bound and verified by the control plane — never self-asserted. A **per-instance** credential is minted here. |
+| **Running** | desired=running ∧ identity_verified ∧ accepting_work ∧ healthy | Alive, authorized, in-sync, and admitting work. | **Only Running admits new work** → dispatch/gate is the single predicate `state == Running`. |
+| **Paused** | desired=running ∧ identity_verified ∧ ¬accepting_work ∧ healthy | Healthy and in-sync but deliberately not admitting (director cordon). | Intent, not fault. Resumable; still subject to health edges. Keeping it a peer state is what keeps the dispatch predicate single-field. |
+| **Unhealthy** | desired=running ∧ identity_verified ∧ ¬healthy | Alive but fenced: liveness/authz/probe/lease lost. **Not** version skew. | Fenced at once; recover within a window (re-prove identity) or go to Stopping. Split cause: *observed-bad* vs *unobservable* (node lost). |
+| **Stopping** | desired=stopped; graceful window open | Terminate committed: flush state and finish in-flight work within a deadline (may still be health-OK). | `desiredStatus==stopped` is the cross-runtime discriminator. Durability was already secured while Running. |
+| **Stopped** | terminal (absorbing) | Terminated. Not resurrected; a replacement is a fresh instance. | Record the cause (normative enum: normal / crash / reclaimed). Granularity is **instance-level**. |
+
+**Attributes, not states** (read alongside the state): `accepting_work`
+(Running vs Paused) — its authority is the **CP/director**, never the agent's
+self-report; `superseded` / version-skew (a healthy instance whose desired
+version has moved on) ⇒ `accepting_work=false`, so it classifies as **Paused**
+and is never dispatched new work. *When and in what order* a superseded instance
+is drained or replaced is a **fleet-level rollout** concern (e.g.
+make-before-break) — out of scope for this instance-level ADR; see the future
+rollout / RuntimeDriver ADR. Also: health `cause` = observed-bad vs
+unobservable; death `cause` enum; turn-level busy/idle.
+
+## 4. Principles
+
+1. **Default-deny identity.** Identity is proven with a control-plane-issued
+   credential, never accepted from the agent's own claim. The **trust root is
+   the runtime's injection primitive** (IRSA / k8s projected SA token) that
+   delegates a platform identity — state it explicitly. **Role identity ≠
+   instance identity**: mint a **per-instance** credential at `Starting`.
+2. **Trust & sync are continuous.** Heartbeat carries a CP-signed, short-TTL
+   **lease token bound to the instance id** (task ARN / pod UID). A **monotonic
+   fencing epoch** guards generations — the CP accepts only the highest epoch,
+   defeating zombie/split-brain after a partition. Credentials are revoked on
+   Stopping/Stopped; `Unhealthy→Running` must re-prove identity.
+3. **Only `Stopped` is terminal (absorbing), at instance granularity.** A
+   container restart within the same pod is the *same* instance, not a
+   `Stopped→Starting` flap; restart = a new lifecycle only when a new instance
+   is created.
+4. **`reclaim` is two paths, not one.** A *planned* interruption (Spot/preempt
+   notice — ECS ~120s SIGTERM, GKE ~30s + preStop) **compresses `Stopping`**
+   into a short deadline. Only a *hard* loss (node death / SIGKILL / OOM) jumps
+   straight to `Stopped`. Durability never relies on the Stopping window —
+   **checkpoint while Running.**
+5. **Runtime-independent.** Each driver projects native signals onto the 6 via
+   the discriminators `(desiredStatus, accepting_work, health, identity_verified)`;
+   the machine never changes per runtime.
+6. **Two predicates, kept apart.** *Dispatch new work* = `state == Running`
+   (single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping`(within
+   deadline) — a cordoned (Paused) agent still finishes its current turn / MCP
+   call. Don't collapse them into one sentence.
+
+## 5. Model: config vs observed
+
+`Instance = Desired Spec (identity + version) + Observed State`. Desired and
+observed are strictly separated; **state is observed, not part of the desired
+config**. "In sync" (Running) means the reconcile loop has zero diff on the
+desired spec. (This replaces the earlier `config = identity + version + state`,
+which folded observed state into desired config and could never reconcile to
+zero diff.)
+
+## 6. Runtime Independence (projection)
+
+Discriminators, not native strings. `desiredStatus==stopped` is one signal
+across runtimes: **ECS `desiredStatus STOPPED` ⟺ k8s `deletionTimestamp!=null`
+⟺ compose stop-requested** — that is what makes `Stopping` decidable rather than
+an ECS-only coincidence.
+
+| canonical | ECS | k8s / GKE | docker-compose |
+|---|---|---|---|
+| Starting | PROVISIONING / PENDING / **ACTIVATING** (ENI + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting |
+| Running | RUNNING + health OK + desiredStatus RUNNING | Running + readinessProbe True + lease valid | healthy *(healthcheck required)* |
+| Paused | RUNNING + health OK + CP/director cordon (`accepting_work=false`) | Ready but cordoned (CP/director) | running + CP/director cordon |
+| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (SIGSTOP) → healthcheck stall → Unhealthy |
+| Stopping | desiredStatus STOPPED *(DEACTIVATING only if in a target group / service-discovery; else RUNNING→STOPPING)* | deletionTimestamp != null (Terminating: preStop + grace) | stop requested (stop_grace_period) |
+| Stopped | STOPPED + stopCode (enum) | deleted; *preempted* = the reclaim edge | exited |
+
+**Driver conformance conditions**
+- A driver must expose all four discriminators (including the latching
+  `identity_verified`); if it cannot, it does not conform.
+- **docker-compose requires a `healthcheck`** — without one it only sees
+  running/exited and can never separate Running from Unhealthy.
+- **docker-compose must set `restart: "no"`** and hand restart to the control
+  plane; `restart: unless-stopped` auto-resurrects a crashed container, which
+  contradicts "Stopped is terminal" and competes with reclaim/replace.
+
+## 7. Considered Options
+
+- **6 states with Paused as a peer state (chosen).** Uses the discriminators to
+  define Paused rigorously; keeps dispatch single-field.
+- **5 states, Paused/Draining as a `Running` attribute** (reviewers' converged
+  proposal) — *rejected as the surface model* because it forces a two-field
+  dispatch predicate (`Running && accepting_work`); every caller that forgets
+  `&& accepting_work` silently mis-schedules a paused agent. **We adopt its
+  `(desiredStatus, accepting_work)` machinery as Paused's definition.**
+- **Hermes' 6 operational states verbatim** — rejected: mixes install/service
+  concerns with runtime state; path/name identity is the self-report we reject.
+- **pi `idle/turn` as the primary machine** — rejected: a sub-layer of Running.
+- **K8s granular phases** (Pending/Running/Succeeded/Failed/Unknown + container
+  states) — rejected for the surface set; folded into attributes.
+- **Drop `Unhealthy`** — rejected: loses the "alive but fenced" distinction.
+
+## 8. Prior Art
+
+| Project | Model | What we take / differ |
+|---|---|---|
+| **Kubernetes** Pod lifecycle | Phase + Conditions + Probes (three-layer decoupling); `Unknown` on node loss | Direct ancestor; we take the phase/condition/probe split; `Unknown`→Unhealthy(fenced). |
+| **HashiCorp Nomad** | alloc states pending/running/complete/failed/**lost**; driver preemption events | `lost`/`unknown` is exactly our *unobservable* Unhealthy case. |
+| **Temporal / Cadence** | workflow/activity states + heartbeat **lease fencing** | Validates the fencing epoch on the heartbeat lease. |
+| **Erlang/OTP supervisor** | child spec + crash exit reason + `one_for_one`; restart spawns a new child | Supports "restart = new lifecycle / fresh instance". |
+| **AWS EC2 instance lifecycle** | pending/running/stopping/stopped/terminated | Near-identical shape; instance-level granularity. |
+| **systemd unit** | active / **failed** / … as first-class | `failed` as a first-class fault state. |
+| **Ray actor** | PENDING / ALIVE / RESTARTING / DEAD | Close 1:1; `RESTARTING` = our replace path. |
+| **Hermes / Pi / Pi-Desktop** | ops CLI states / in-process turn engine / desktop shell | Adjacent code, not instance-level lifecycle. Pi validates **checkpoint-while-Running**. |
+
+## 9. Consequences
+
+- The read-model and Studio report **only these 6 states**.
+- Every runtime driver must provide a **native→6 projection** via the
+  discriminators (conformance requirement), including the compose healthcheck
+  and `restart:"no"` conditions above.
+- Detailed sub-states are **attributes** of the 6 (accepting_work, superseded,
+  health-cause, death-cause enum, busy/idle), not new states.
+- **Follow-ups:** a `RuntimeDriver` contract ADR (verbs apply / observe / scale
+  / cordon / …); an identity / lease / epoch spec ADR.
+
+## 10. More Information
+
+Format follows **MADR** (markdown ADR: context → drivers → options → decision →
+consequences) with a **Nygard** status/context/decision/consequences spine and a
+**Y-statement** summary. See `docs/review-runbook.md` for the review rubric this
+ADR was gated on.

docs/review-runbook.md

@@ -0,0 +1,59 @@
+# Review Runbook
+
+How we review ADRs and design docs in this repo. The goal is a **falsifiable**
+review — reviewers try to break each load-bearing claim, not nod at it. Peer
+"LGTM" carries weight only after the claim has survived an attempt to refute it.
+
+## The 8 axes
+
+Every **load-bearing claim** in a doc is scored against all 8. A claim survives
+only if it passes **every** axis.
+
+1. **Simplicity / concise** — minimal surface area; no state/column/sentence that
+   could be cut. *Fails on:* bloat.
+2. **In scope** — decides only what this doc is for; no sprawl (e.g. don't fold a
+   RuntimeDriver contract or implementation detail into a state-model ADR).
+3. **Factcheck** — runtime behaviour and prior-art claims are true, **with a
+   source**. A claim with no source does not pass.
+4. **Refute** — assume the claim is *wrong* and try to prove it (adversarial
+   default); it survives only if the refutation fails.
+5. **Coverage / MECE** — exhaustive and mutually exclusive. Ask "what state /
+   edge / runtime situation is missing?" and "can one situation fall into two?"
+   (Distinct from Refute: Refute attacks "what you said is wrong"; Coverage
+   attacks "you didn't say X".)
+6. **Consistency** — sections don't contradict each other (definition ↔ diagram
+   ↔ projection ↔ principles) and align with the doc's own first-principles.
+7. **Decidable / actionable** — the decision is actually made, and an
+   implementer/driver can act on it without ambiguity.
+8. **Reversibility / lock-in** — what this locks in and how expensive it is to
+   change later.
+
+## Verdict rule
+
+- Score each load-bearing claim across all 8 axes.
+- **Refute** defaults to *refuted* — a claim is only "survived" once refutation
+  attempts fail.
+- **Factcheck** with no source does not pass.
+- Report only the axes a claim **fails**, with the counter-example or source.
+  Passing axes need no restatement.
+
+## How to run a refute pass
+
+1. Enumerate the doc's load-bearing claims (the ones the decision rests on).
+2. Assign refuters; each is told to assume the claim is wrong and produce a
+   counter-example, a missing case, or a contradicting source.
+3. A claim survives only if no refuter lands. Surviving-with-fixes → fold the
+   fix; failed → back to the author.
+4. Consolidate into one review comment on the PR; the author decides how to land.
+
+## References (ADR writing)
+
+- **Michael Nygard**, *Documenting Architecture Decisions* — the origin;
+  Status / Context / Decision / Consequences.
+- **MADR** — Markdown ADR: context → drivers → considered options → decision
+  outcome → consequences. <https://adr.github.io/madr/>
+- **adr.github.io** — templates and `adr-tools`. <https://adr.github.io/>
+- **Joel Parker Henderson**, ADR templates & examples collection.
+  <https://github.com/joelparkerhenderson/architecture-decision-record>
+- **Y-statement** — one-line decision summary: "In context X, facing Y, we
+  decided Z, to achieve W, accepting V."