Document Backup Restore

Document backup: revert and drill

How to put documents back from the Azure archive backup, and how to run the restore drill. The account, access rules and backup run are in Document-Backup.

Measured 2026-10-06, test only. The revert tool has not written to any bucket. Its read-only steps (plan, status, and the dry runs of rehydrate and apply) ran against the test account; the writes are covered by unit tests with fake stores only. The drill passed on test: 20 of 20 copies matched. Any command marked "not yet run as written" has not been run on a real environment.

The object store honours the Content-MD5 header the tool relies on: a wrong value returns 400 InvalidDigest, tested against the test bucket on 2026-10-05. The tool does not send If-Match, because the NRS store answers 412 to it on objects stored years ago, even when the ETag matches. Each write is verified afterwards by reading the object back.

Revert documents to the backup

The revert tool, src/scripts/restore-originals.js, copies the original from the archive back over the bucket object and marks each document's pdfTitle record restored. It has four modes, and only the last one writes. Each mode is a dry run unless given --live.

Before starting:

• Stop the PDF title worker. On test, disable the function pdf_title_run on demi-pdf-title-test (or set PDF_TITLE_LIVE off), then wait for the running tick to finish; a tick takes about 3 minutes. Prod has no worker yet. Re-enable the worker after the restore. The tool re-reads each document's lease before it writes and refuses a leased row.
• Blob storage and Cosmos need different identities, and no one identity holds both. Cosmos clients sign in as AZURE_CLIENT_ID. Blob storage clients sign in as BACKUP_CLIENT_ID when it is set, and as AZURE_CLIENT_ID when it is not. restore-originals.js shares this with backup-originals.js (defaultContainers). Tested on test 2026-10-06 with one identity: as demi-identity-<env> (the devbox default), plan works but status fails on restore, where that identity has no role; as the writer, plan stops with a Cosmos RBAC error. With Cosmos as demi-identity-<env> and blob storage as the writer in one process, plan, status and the rehydrate and apply dry runs all worked.
• Run shape on test: AZURE_CLIENT_ID is the client id of demi-identity-test, BACKUP_CLIENT_ID is the client id of eagle-backup-writer-test, during a grantWriter=true window, from a VM that has both identities attached. Live rehydrate and apply have not been run with this shape. Take the object-store settings from the backup env file (see "Run a backup" in Document-Backup).
• The tool needs STORAGE_BACKEND=minio and an empty MINIO_KEY_PREFIX, because blob names are the stored s3Key. It refuses to run otherwise.

Choose the documents with exactly one of --id <docId>, --ids <file> (one id per line), --project <projectId> or --all-changed (every document, keeping rows that need work). A row is one bucket key with the selected documents that store it. The examples use --ids ids.txt. Every mode also takes --max N, --concurrency N (default 8), --container and --restore-container.

1. Download the manifest of the verify run and run plan. The manifest path in manifests is <env>/<runId>/objects.jsonl.gz. The tool logs the manifest's sha256; compare it with the manifest line in summary.json. plan --id ran on test 2026-10-06 as demi-identity-test (decision unchanged, exit 0, logged sha256 equal to summary.json). The manifest was fetched with the storage SDK; the az storage blob download line and --all-changed are not yet run as written:

   az storage blob download --auth-mode login --account-name <account> -c manifests -n <env>/<runId>/objects.jsonl.gz -f objects.jsonl.gz
   node src/scripts/restore-originals.js plan --all-changed --manifest objects.jsonl.gz --out ids.txt
   

   plan is read-only. It logs one line per row with a decision. --out saves the ids of the rows that need work, so later steps act on the same set.

   Decision	Meaning
   restore	The bucket object differs from the backup, by size, MD5 or ETag
   record-only	The object already matches but a record still says titled. A rerun finishes a run that stopped after the write
   unchanged	Nothing to do
   refused	The row is not touched. Reasons include no-backup, backup-not-verified:<status>, missing-in-bucket, a lease or in-flight title, and record-original-differs

2. Rehydrate. Archived blobs cannot be read. This copies each restore row into the restore container at Cool tier, standard priority. Run it without --live first, then with it. The dry run worked on test 2026-10-06 (outcome planned) with the split identities above; --live is not yet run as written:

   node src/scripts/restore-originals.js rehydrate --ids ids.txt --manifest objects.jsonl.gz --account <account> --live
   

   Rerunning is safe: pending and ready copies are skipped and failed ones start again.

3. Wait up to 15 hours, then check. status counts rows as absent, pending, ready and failed. It exits 2 while any row is not ready. On test 2026-10-06 it counted a row absent and exited 2 with the split identities; as demi-identity-test alone it failed the row. Not yet run as written with BACKUP_CLIENT_ID set:

   node src/scripts/restore-originals.js status --ids ids.txt --manifest objects.jsonl.gz --account <account>
   

4. Apply. The dry run prints would write N rows; rerun with --live --confirm N. Run it, then repeat with --live --confirm N, using that N. A missing or different count writes nothing and exits 1. The dry run on test 2026-10-06 held a not-yet-rehydrated row as pending and exited 2. --live is not yet run as written:

   node src/scripts/restore-originals.js apply --ids ids.txt --manifest objects.jsonl.gz --account <account>
   node src/scripts/restore-originals.js apply --ids ids.txt --manifest objects.jsonl.gz --account <account> --live --confirm <N>
   

   Rows are written one at a time. For each row the tool:

   1. Downloads the rehydrated blob and checks its size, MD5 and sha256 against the manifest.
   2. Re-reads the documents and skips the row if one has been leased since the plan.
   3. Reads the object's ETag just before the write. If it differs from the ETag seen at plan time, the row is reported as changed and skipped. Otherwise it sends a signed PUT with a Content-MD5 header and no If-Match, because the store rejects If-Match on older objects.
   4. Reads the whole object back and checks size, MD5 and sha256. A mismatch is failed: read-back-mismatch and the record is left alone.
   5. Updates each stale record: pdfTitle.status is restored, reason is archive-restore, title, titledLength and titledSha256 are null, and at is set. The original fields are kept.

5. Ask the owner to remove the grants from the identity.

Exit codes: 0 clean; 1 any row refused, changed or failed, or an id not found; 2 rows still rehydrating. Each row logs [restore] <mode> key=<key> docs=<ids> outcome=<o> reason=<r>, then a summary line.

Nothing is deleted. The tool never deletes from the bucket or the archive. Both buckets have versioning enabled and expire older versions after 60 days (bucket lifecycle setting, read 2026-10-05), so a replaced file stays recoverable from the bucket for 60 days. The rehydrated copies in restore are removed by the lifecycle rule 30 days after creation.

Limits of the revert tool

• Rehydration takes up to 15 hours at standard priority. The tool does not offer high priority; its price was not checked.
• It refuses a row whose object is missing from the bucket, and a row whose record has an in-flight title.
• The restored status offers the file for titling again if the document is eligible.
• Blob storage uses BACKUP_CLIENT_ID and Cosmos uses AZURE_CLIENT_ID; see "Before starting".

Cost: about $0.001 for a 3 MB file. A full restore of 1 TB is about $22.50 for the read-back and about $82 in operations, plus Cool storage while the copies are held. Egress back to the object store is not priced.

Restore drill

The drill proves that archived content comes back intact. It copies a random sample of backup blobs into restore, waits for rehydration, compares each file's sha256 with its metadata, and deletes the copies. Use 20 blobs on test and 10 on prod. It runs as the writer identity (grant steps are in Document-Backup), in two phases up to 15 hours apart, so the grant must stay on until the check has run: the check reads and deletes in restore.

1. Start. Writes the list to --drill-list (live only) and starts the copies. This is the command the test run used:

   node src/scripts/backup-originals.js drill --account $ACCT --drill-list /home/demi/backup-run/drill.jsonl --sample 20 --live
   

2. Check, after rehydration ends. With --live it deletes each copy that matches. Exit 0 means all matched, 2 means some are still rehydrating (rerun later), 1 means a mismatch or a failed copy. Passed on test 2026-10-06, about 22 hours after the start: 20 matched, 0 mismatch, 0 pending, 0 failed, and the 20 copies were deleted. The first attempt, a minute after the grant was deployed, failed all 20 because the role had not reached the storage service yet; a rerun two minutes later passed:

   node src/scripts/backup-originals.js drill --account $ACCT --drill-list /home/demi/backup-run/drill.jsonl --check --live
   

Copies left in restore are deleted by the lifecycle rule 30 days after creation.