feat(hooks): offline payload spool + drain — buffer unsent events, replay on recovery

[daviddanialy]

Part of DNO-498 (DCPD 4) — part 1 of 2: capture. Today a payload whose send fails against a down control plane dies with the hook process; this PR spools it for replay so the event still gets scanned/flagged/monitored retroactively. Placement rationale (hooks own the buffer; the device agent supplies the recovery trigger) is in device-agent ADR-0012; plan and decisions are on the Linear ticket.

What

• relay/spool.go — when an exchange ends unsent (transport failure, 5xx, or 429/408 — the server didn't store the event; a definitive 4xx never spools), the built IngestRequestBody is persisted under $XDG_STATE_HOME/gram/hooks/spool/. One JSON file per entry, named <zero-padded-nanos>-<pid>-<seq>.json so lexicographic order is chronological order and concurrent hook processes never contend on a write — load-bearing because the existing file lock is a no-op on Windows (codextools_lock_windows.go).
• Entry contents: the envelope (whose occurred_at was set at build time, so replay preserves original timestamps), the send's idempotency key, and the deployment identity (server_url/org_id/project_slug/config_path) a drain needs to route and re-resolve auth per entry.
• Idempotency key threading — the key is now minted in deliver and passed through client.send (both the first attempt and the org-key auth retry reuse it). This is what makes every redelivery path safe: SDK-internal replays, the org retry, and a future drain of a spooled copy all dedupe server-side against any partially delivered original.
• Caps enforced at write time: 2,000 entries / 64 MiB, drop-oldest with a debug-logged count, plus 14-day expiry (mirroring the org-settings cache ceiling). Spooling is best-effort by design — a spool failure only logs; the gating outcome was already decided by the ratchet and fail open/closed, and buffering never touches the session path.
• The ratchet still runs first: never-authenticated machines send nothing and therefore spool nothing.

Nothing reads the spool yet — this ships safely alone (bounded, aging disk usage; zero behavior change on the wire). Part 2 adds the top-level speakeasy-hooks drain subcommand (replay oldest-first with stored keys, X-Gram-Replayed: 1 marker) and the opportunistic detached drain after a successful send; the device agent's recovery trigger (merged, dormant) will exec it on agent-managed machines.

Tests

• Full provider-path capture test against a dead control plane (entry contents, identity fields, session round-trip), plus negative paths: 200/4xx answers don't spool, unauthenticated machines don't spool.
• Wire test pinning that client.send uses the caller's idempotency key verbatim.
• Trim tests: count cap, bytes cap, age expiry, foreign files left alone; filename-ordering and unsent predicate tables.
• go test -race ./... green across hooks packages; CGO_ENABLED=0 builds for darwin/linux/windows.

🤖 Generated with Claude Code

---

Part 2 (second commit): drain

• speakeasy-hooks drain — top-level subcommand (dispatched in main.go alongside version/login/install; the external agenthooks lib is untouched). Takes no arguments: entries carry their own deployment identity. Replays oldest-first (global chronological order ⇒ per-session order), each request under its originally minted Idempotency-Key plus X-Gram-Replayed: 1 so the server can distinguish downtime backlog from live traffic (DNO-499 contract).
• Per-entry outcomes: accepted → delete; definitive 4xx or corrupt file → drop (a replay would fail identically / could never parse); unreachable/5xx/429/408 → abort the run and keep the backlog (the server is down again — don't hammer). Entries past 14 days expire unsent. Entries with a newer schema version are left for a newer binary.
• Auth at drain time re-resolves per deployment exactly as a live send would (env key → disk cache → the recorded config file's org key), memoized per identity; insecure server URLs are refused, matching the live-send guard.
• Self-healing without the agent: the first successful send while entries exist spawns a detached drain (debounced 30s), so hooks-only installs flush on the next natural hook event. The device agent's recovery trigger (device-agent PR chore(deps): bump k8s.io/apimachinery from 0.33.3 to 0.33.4 in /server #72, merged) execs the same subcommand the moment its detector sees recovery — covering idle machines.
• Exit code 1 when entries remain or the run aborted, so the daemon's log shows incomplete runs.

Drain tests cover: ordering + stored-key + replay-marker wire assertions, abort-and-keep on a dead server, 4xx drop, expiry-without-send, newer-schema skip, corrupt-entry drop, credential-less skip (exit 1), config org-key fallback (asserted on the wire), spawn-on-recovery with debounce, no spawn on failed sends or empty spools, and exit codes. The invoke test helper is now hermetic w.r.t. XDG_STATE_HOME so the suite can never touch (or drain) a developer's real spool.

[linear-code]

DNO-498

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2d90d8d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

cubic analysis

All reported issues were addressed across 5 files

Linked issue analysis

Linked issue: DNO-498: DCPD 4: Buffer session/tool-call payloads offline and replay on recovery

Status	Acceptance criteria	Notes
✅	Durable local on-disk queue for unsent hook payloads (path under $XDG_STATE_HOME/gram/hooks/spool)	The PR creates a spool directory, persists one JSON file per unsent entry, and ensures concurrency-safe filenames.
✅	Queue caps and retention (count cap, bytes cap, age expiry) with drop-oldest behavior and logging when entries are dropped	Caps are enforced at write time (entry and byte caps), oldest entries are dropped first, 14-day expiry is implemented, and debug logs emitted on drops; trim behavior is covered by tests.
✅	Preserve original event timestamps and maintain chronological ordering in the spool	The envelope carries the occurred_at set at build time and entries are written with timestamp-prefixed filenames whose lexicographic order is chronological; tests verify filename ordering and that spooled Envelope retains event fields.
✅	Server-side replay safety via idempotency/dedupe (idempotency key minted and threaded through send and persisted with spooled entries)	An idempotency key is minted in deliver, passed into client.send for all attempts, and stored in the spool entry so future replays will use the original key and dedupe server-side; tests assert the wire header uses the caller's key.
✅	Decision on buffer placement (hooks-owned spool with device-agent supplying recovery trigger)	The PR implements the spool inside hooks and documents the placement rationale; it explicitly defers replay/drain to part 2 and references the device-agent ADR for the recovery trigger.

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 9 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 11 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>