feat(catalog): archive retired agents out of the live catalog

[schickling-assistant]

Problem

desired-state "retired" is runtime teardown only — the spec and resources/ stay byte-identical and reversible (#439, dotfiles#1535). That is the right semantics, and it means a long-lived catalog accumulates retired identities forever: dev3 holds 655 retired specs today, resources/ is >99% of their bytes, and st2 catalog graph --json reports 696 declarations. The cost is catalog size — discovery, validation, reconcile and doctor pass over every one of them — not disk. There was no way to take an identity out of the live catalog without deleting it and losing the trace.

Decided in Q3 (seat dev3.direct.omp.v6c4mkm2, option retired-keeps-all-plus-archived): lifecycle becomes running → suspended → retired → archived; archived moves the identity out of the live catalog with a tombstone, and un-archive is the reverse move under the same lock.

Change

st2 catalog archive [--identity <id>]... [--all-retired] [--host <h>] [--dry-run] [--json] and st2 catalog unarchive <id> [--host <h>] [--json], in a new src/catalog_archive.rs (no new crate).

• Where. <catalog>/.st2/archive/<host>/<identity>/. .st2 is control space at any depth in agent_spec::discovery::is_catalog_path, and catalog_transaction's projection only walks agents/, _templates/, catalog.kdl and profile modules — so an archived declaration is structurally undiscoverable and never enters a whole-catalog transaction's declaration identity. No new exclusion rule was added anywhere.
• How. Exclusive CatalogLock + one begin_generation_commit() around the batch; fs::rename plus sync_dir on both parents. The archive root is a child of the catalog root, so the move is same-filesystem by construction — there is deliberately no copy+verify+remove engine for a case that cannot arise; an EXDEV rename surfaces as an error naming the constraint. The rename lands before the tombstone write on purpose: a crash in that window leaves an archived directory the graph reports and unarchive still reverses, whereas tombstone-first would advertise an identity that never moved.
• Eligibility, fail-closed. Discovery must be error-free (otherwise a supervisor reference could be hidden); the declaration must sit at its canonical agents/<host>/<identity>/agent.kdl; be retired in either spelling; have no live or dead runtime record for any declared task — the same SystemRunner::list_sessions() rule st2 doctor applies to retirement; and be named as supervisor by no declaration that stays behind (a retired dependent leaving in the same --all-retired run is not a live reference). Local host only: another host's runtime records are not observable from here, matching doctor's own host filter. Spec-less directories are out of scope (Q4).
• Semantics. --identity refuses the whole run if any named identity is ineligible (nothing moves); --all-retired reports the ineligible ones in refused and archives the rest. --dry-run decides without creating the archive root.
• Graph. Additive archived: [{id, host, identity, archivedAt, reason, archiveRoot}] on the existing st2.catalog-graph.v2 envelope — no schema bump, no field removed, no shape change for existing readers. An ordinary archived identity leaves complete: true untouched; an archived directory with no readable tombstone (or a tombstone with no directory) is unexplained control-plane state and yields an archive-unexplained error issue with complete: false.
• Retirement spelling. Already collapsed on the authoring write path: agent_author::desired_state_edit treats a legacy retired #true node as the lifecycle node and replaces it with desired-state "…" reason=… (or removes it for running). Proven by tests/agent_desired_state.rs::cli_canonicalizes_legacy_retirement_and_refuses_nix_owned_declarations and ::legacy_retirement_reads_and_authoring_preserves_resources; I wrote a third test for it and deleted it as duplication (Q9). Deliberate deviation from the brief: agent publish / catalog apply do not rewrite caller bytes to the canonical spelling, because those are content-addressed — normalizing on the way in would break --input-sha256/--expect-sha256, the exact CAS the transaction is built on. Read compatibility stays; legacy declarations converge through the authoring verb and drain out through archive, and archive accepts both spellings.

Docs: README (workflow + command surface), docs/vrs/spec.md (R35 catalog-graph area), one new INVARIANTS row Archival leaves the live catalog with seven named proofs.

Gates and evidence

Base is schickling-assistant/2026-09-04-b2-retired-keeps-resources (#439), which is open and unmerged at time of writing — this PR is stacked on it, not on main. #439's branch itself is based on 5b2ee7c, one merge behind main.

cargo test --all-targets --all-features, the 12 new tests in tests/catalog_archive.rs all green:

test	proves
archive_moves_a_retired_seat_out_of_discovery_with_its_resources_byte_identical	spec + resources/ (incl. resources/inbox/) byte-identical after the move; agents/… gone; st2 ls sees nothing; tombstone written; graph has one archived row, complete: true
archive_accepts_the_legacy_retirement_spelling	retired #true archives, reason is null
archive_refuses_a_running_or_suspended_declaration	not-retired, nothing moves, archive root never created
archive_refuses_while_any_declared_task_record_survives	runtime-record-present for both an alive and a dead record
archive_refuses_an_identity_another_declaration_still_names_as_supervisor	supervisor-referenced, names the dependent
dry_run_reports_the_plan_and_changes_nothing	plan emitted, archivedAt: 0, no archive root
all_retired_archives_every_eligible_seat_and_reports_the_rest	sweep archives the eligible one, reports the blocked one, exits 0
all_retired_archives_a_retired_supervisor_together_with_its_retired_dependent	a dependent leaving in the same run is not a live reference
unarchive_restores_the_identity_byte_identically_and_clears_its_tombstone	round trip, tombstone gone, graph row gone, agent row back
an_archived_directory_without_a_tombstone_makes_the_graph_incomplete	archive-unexplained error issue, exit 1
archive_refuses_an_unknown_identity_and_a_second_archive_of_the_same_identity	unknown-identity; the archived copy is never silently replaced
unarchive_refuses_to_overwrite_a_live_declaration	refuses rather than clobbering

Live smoke run of the built binary against a temp catalog (--help, --dry-run, archive, find, graph JSON, refusal of a running agent, unarchive) — all as documented; output shape:

would-archive dev.old agents/dev/old -> .st2/archive/dev/old
archived      dev.old agents/dev/old -> .st2/archive/dev/old
archived [{'id': 'dev.old', 'archivedAt': 1788532451235, 'reason': 'Migration finished', 'archiveRoot': '.st2/archive/dev/old'}]
Error: refusing to archive dev.lead: [not-retired] desired state is 'running'; archive requires 'retired'
unarchived dev.old .st2/archive/dev/old -> agents/dev/old

rustfmt --check is clean on both new files and on every hunk I added to src/main.rs, src/catalog_graph.rs and src/lib.rs. I did not reformat the pre-existing lines rustfmt also wants to change in those files.

Pre-existing red, unchanged by this PR

Proven by stashing this diff and re-running on the base commit:

• tests/agent_publish.rs — the same 9 failures before and after (cas_rejects_stale_writers_and_preserves_resources, concurrent_publishers_serialize_and_only_one_wins_the_cas, control_directory_swap_cannot_redirect_publication_staging, intermediate_host_swap_cannot_redirect_publication_outside_the_catalog, publish_post_commit_generation_failure_is_fenced_and_recovered, retirement_cannot_commit_between_reconcile_discovery_and_launch, spec_publish_crash_stages_only_in_the_control_plane, success_receipt_requires_exact_locked_readback, success_receipt_requires_locked_full_catalog_readmission). All fail on candidate fails full-catalog validation … host 'host' must declare exactly one root agent; found 0 — those fixtures publish a sole retired #true agent, which the Graph v2 counts legacy-retired declarations as roots #402 root-count fold no longer counts as a root.
• tests/invariants.rs::qualified_proof_references_resolve — panics on tests/resource_profile_supervisor_e2e.rs::demand_observation_settlement_matrix_is_atomic_and_preserves_facts, a proof reference introduced by Add atomic Resource demand observations #404 that names a test that does not exist on main either. My seven new proof references were verified to resolve by hand (the assertion aborts on the first bad reference, so it never reaches them).
• cargo clippy --all-targets --all-features -- -D warnings — fails to compile crates/st2-resource-protocol on manual_is_multiple_of at src/lib.rs:410, untouched here. Scoped clippy reports zero findings in src/catalog_archive.rs and tests/catalog_archive.rs.
• cargo fmt -- --check is red tree-wide on main (agent-spec examples, discovery.rs, resync.rs, run.rs, resource_profile.rs, …).

Intentionally left

• No copy+verify+remove fallback. The archive root is inside the catalog root; a cross-device rename cannot happen without someone mounting a filesystem mid-catalog. A bespoke copy engine for that case adds global complexity (Q1) and a partially-copied-bundle failure mode that does not otherwise exist. EXDEV surfaces as an error naming the constraint.
• No normalization inside publish / catalog apply. Content-addressed; see above.
• No validate warning for the legacy spelling. It would add ~655 warnings to dev3's graph issues (already 364) for a form that read compatibility keeps supporting and that archive drains.
• Cross-host archival. Refused: this host cannot observe another host's PTY/exec records, so it cannot prove runtime-record absence. Same boundary st2 doctor already draws.
• Spec-less catalog directories (direct.* with no agent.kdl) — out of scope per Q4; operating only on discovered specs excludes them automatically.
• No deployment. Running archive on dev3 belongs to B4 (dotfiles#2410, dotfiles#2411).

Refs: schickling/dotfiles#2408, schickling/dotfiles#2411, decision Q3 smkfdt.

Posted on behalf of @schickling

field	value
agent_identity	dev3.direct.omp.v6c4mkm2
session	dev3.v6c4mkm2
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.1.2
agent_runtime	OMP 18.1.2
tooling_profile	dotfiles@7534055