recovery: destination reset + machine-replacement runbook (F20, F28)

[mbertschler]

Summary

The two recovery scenarios the reference-setup walk dead-ended on (friction log F20 + F28). Both are now explicit, human-driven, audit-preserving acts — the agent never runs them on a cadence.

Stacked on #162 (PR #171); base retargets to main once #162 merges. This PR leans on #162's node pair helper for the machine-replacement runbook, so its base is claude/issue-162-bootstrap-helpers to keep the diff to just this delta. GitHub retargets it to main automatically when #171 merges.

Changes

• squirrel destination reset <name> (F20) — forgets a destination's recorded remote state (remote_objects/remote_packs, the durability vector, push-freshness) so the next sync re-uploads. A change command per ux-principle 2: prints what it will clear, refuses without --yes, offers --dry-run, reports loudly with the run id. A configured-but-unrecorded destination reports "nothing to reset" rather than minting an empty run.
• Empty-root fresh-start recognition (F20) — the packed and content-addressed layout guards keyed on run history by destination name, so a fresh root still refused. Now, when the layout marker is missing, the guard probes the configured root: an empty root (wiped/repointed, or cleared by destination reset) is a fresh start; a non-empty root or any probe error keeps the refusal (fail-closed). The refusal message points at destination reset.
• Machine-replacement runbook (F28) — new docs/guides/recovery.md documents rebuilding a dead edge machine from its hub via a reverse peer push (the same mechanism nas→htpc uses daily), re-pairing with cli: day-1 bootstrap helpers — config check, agent cert, node pairing #162's node pair. restore's node dead-end error now signposts it instead of just refusing.
• Cross-references the manual DR that already works and stays: mirror restore, ride-along index-snapshot recovery (checkpoint 8), and packed/content-addressed recovery.

Audit-preserving guarantee: the reset never touches the runs table, the append-only destination_run_ids_history advance log, or the content/files rows. Only derived state is cleared, and the reset is itself recorded as a run.

⚠️ No schema change needed (confirming, per the schema-chain caution)

runs.kind has a CHECK constraint, so a new run kind would need a migration — the parallel #157/#158 chain owns v25/v26, so I did not add one. Instead the reset reuses the existing audit run kind (the destination-scoped, volume/destination-NULL shape BeginRemoteVerifyRun already uses) and records its detail via a new free-text runs_audit transition reset-destination (that column has no CHECK — new transitions need no migration). SchemaVersion is unchanged; TestSchemaSnapshot stays green.

⚠️ Design decisions to confirm

1. Destination-reset command shape. squirrel destination reset <name> with --yes (confirm) + --dry-run. Bare (no --yes) prints the preview and refuses. It clears derived state only and leaves the remote bytes untouched — the operator wipes/repoints the remote separately, after which the empty-root guard does the rest. Alternative shapes not taken: an interactive y/n prompt (the codebase's idiom is explicit flags like --force/--allow-rewind), or also deleting the remote bytes (kept out — reset is about squirrel's record, and deleting remote data is a heavier, separate act).

2. Machine-replacement approach: reverse peer push (runbook), not restore --from <node>. I chose to reuse the existing peer-sync push driven from the hub rather than build restore-from-node. Rationale: the machinery already exists and runs daily (nas→htpc), it is content-verified end-to-end (peer-blake3) and records correlated runs on both sides, and re-pairing reuses #162's helper — zero new transfer code, lowest risk. restore --from <node> would mean pulling a volume down over a protocol path that doesn't exist today. The only code change on this front is turning the restore dead-end into a signpost. Please confirm you'd rather have the documented reverse-push than a new restore --from <node> verb.

Testing

• go vet ./..., go test ./... — all pass.
• golangci-lint run (v2.12.2, rebuilt with the module's Go 1.26.1 toolchain to clear a packaged-binary version skew) — 0 issues.
• New tests: store reset (clears derived state, preserves runs + advance log + other destinations + content; records the audit run with both transitions; empty/blank-name guards); CLI reset (unknown dest, nothing-to-reset, dry-run, confirmation gate, confirmed clear); sync empty-root fresh start on both layouts (+ the mirror-era guard tests reworked to exercise a genuine non-empty layout conflict). The fake-rclone shim learns lsf.

Scope

Closes #163. Recovery paths only; the manual DR paths are cross-referenced, not reimplemented (packed-native restore remains tracked by #142/PR #169).

🤖 Generated with Claude Code

https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7

---

Generated by Claude Code

[Copilot]

Pull request overview

Adds explicit, audit-preserving recovery workflows to Squirrel: a new operator-driven destination reset command (to clear only derived remote state and force a re-upload), plus “empty remote root ⇒ fresh start” recognition for content-addressed/packed layout guards, and a machine-replacement runbook with improved restore signposting.

Changes:

• Add squirrel destination reset <destination> [--yes] [--dry-run] and store support to clear derived destination state while recording an audit run.
• Teach packed and content-addressed layout watermark guards to treat an empty configured remote root as a fresh start when markers are missing.
• Add recovery documentation/runbooks and update CLI/docs navigation and restore messaging to point users at the supported machine-replacement path.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sync/sync.go	Adds freshStartOnEmptyRoot helper to gate “missing marker” cases.
sync/rclone.go	Adds remoteRootEmpty probe used by layout guards.
sync/packed.go	Uses empty-root probe to allow fresh-start instead of refusing on missing placement map.
sync/packed_test.go	Adds/adjusts tests for mirror-era refusal and fresh-start-on-empty-root.
sync/content_addressed.go	Uses empty-root probe to allow fresh-start instead of refusing on missing manifest segment.
sync/content_addressed_test.go	Extends fake-rclone shim for lsf; adds mirror-era refusal and fresh-start tests.
store/runs_audit.go	Adds reset-destination transition constant for audit trail.
store/destination_reset.go	Implements destination derived-state counting + transactional reset with audit run.
store/destination_reset_test.go	Verifies reset clears only derived state, preserves audit history, and records audit run.
cmd/squirrel/root.go	Registers new destination command group.
cmd/squirrel/destination.go	Adds squirrel destination parent command.
cmd/squirrel/destination_reset.go	Implements CLI UX for reset (preview/refuse unless confirmed, dry-run).
cmd/squirrel/destination_reset_test.go	Adds CLI tests for unknown/nothing-to-reset/dry-run/confirmation/confirmed behaviors.
cmd/squirrel/restore.go	Improves error message when sync_to names a peer node (signposts runbook).
docs/src/content/docs/reference/cli.md	Documents squirrel destination reset and links to recovery guide.
docs/src/content/docs/guides/recovery.md	Adds recovery & disaster runbooks (destination reset + machine replacement + manual DR links).
docs/astro.config.mjs	Adds recovery guide to sidebar navigation.
design/friction-log.md	Updates F20/F28 entries to reference the new supported paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.