Give the event store a restore path, and prove it by destroying one

[xmap]

Why

CORA is described as the system of record for the experiment, and the Postgres event store is where that record lives. There was no backup of any kind: no tool, no WAL archive, no restore path. That is the one claim about CORA a facility reviewer can disprove in a single question, so it blocks the 2-BM pilot ahead of the modelling backlog.

The decision

Closes the Backup / PITR row in docs/stack/deferred.md. pgBackRest. Managed Postgres is unavailable on-premises, and between the two remaining candidates the deciding difference is who runs the restore: a beamline scientist at an awkward hour. pgBackRest generates the recovery configuration itself; WAL-G asks that person to hand-author recovery.signal and restore_command. pg_dump was not a candidate, being a snapshot, so it cannot answer a point-in-time row at all.

The repository TARGET stays deferred deliberately, and cheaply: repo1-type is one config key and no operator-facing command changes with it.

The drill is the point

scripts/restore_drill.py deletes the contents of a real PGDATA and brings it back, verifying at the domain level rather than at "Postgres started".

Run A  ->  full backup  ->  Run B  ->  [target]  ->  Run C

Run B is recoverable only from archived WAL, so it separates a working archive from a working backup. Run C must not return, which separates point-in-time recovery from restore. Each surviving stream's version-ordered fold digest is compared across the restore, so an event dropped or reordered inside a stream fails even with the row count intact.

It also asserts that Run C's WAL segment reached the repository BEFORE the restore, since otherwise its absence would prove a truncated archive rather than a working recovery target.

Executed 2026-07-28 against pgvector/pgvector:pg18 and pgBackRest 2.58.0: 8 of 8 checks pass, schema at revision 20260713000000.

Executing it, rather than writing it, is what found the defects

• The documented restore command aborted on any populated PGDATA (ERROR: [040]), which is every real restore. Needed --delta. The drill now runs the identical command so the two cannot drift.
• Backup and stanza commands need a live cluster connection, not just the data directory.
• pgBackRest selects a backup set at whole-second resolution, strictly-less-than.
• Archiving failures before stanza-create are expected, which makes a naive failed_count = 0 health check useless.

Gate review (4 lenses) also forced an evidence-honesty correction: only 3 of the 8 checks can actually fail in a state the drill constructs. The doc now says which three carry the claim. It further caught position described as the projection watermark, which this repo's own event_store.py contradicts (the cursor is the (transaction_id, position) tuple).

Limits, all stated in the doc

Local disk does not survive host loss. Nothing schedules backups, so nothing expires them either and the repository grows unbounded. Nothing alerts on a failing archive and nothing runs verify. The drill proves the mechanism, not 2-BM's host.

One deferred item has a hidden deadline, now recorded: repository encryption is fixed at stanza creation and cannot be added to an existing repository, so it must be decided before the first backup at any shared or off-site target.

Also records the interaction with out-of-band Atlas migrations: nothing asserts the schema version at boot, so restoring a pre-migration backup under a newer image fails silently. The restore procedure carries that check until the application does.

Test plan

• make restore-drill (executed, 8/8; CORA_DRILL_PORT override available)
• uv run pytest tests/architecture (29601 passed, 590 skipped)
• make docs-build (strict, clean)
• ruff check + format clean on the new script

🤖 Generated with Claude Code

[github-actions]

Coverage report

This PR does not seem to contain any modification to coverable code.