feat(ops): database backups, ADR 0017 (two-leg: nightly pg_dump + encrypted Drive copy)#112
Merged
Merged
Conversation
Nightly pg_dump -Fc of the cercol database kept on-box (7 dumps, /var/backups/cercol) plus a gpg-encrypted copy pushed to Google Drive via rclone (30-day retention). Script exits non-zero on any failure, logs to /home/cercol/logs/db-backup.log and maintains a FAILED marker file, breaking with the silent MAILTO="" cron pattern. Cron file ships in the repo and is installed manually by the operator (same pattern as the SEO ingest crons). Runbook gains the install block, on-box and off-box restore procedures, a quarterly scratch-DB restore test, and loses the backup TODO at the old line 353. CI gains a shellcheck job over api/deploy/backup/*.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolved during implementation: gpg symmetric AES256 (preinstalled; age is not), rclone interactive OAuth token flow (user-owned Drive quota, no service account), retention 7 local / 30 days remote, cron daily 03:15 UTC offset from existing jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_infra.py requires every ADR to carry a Related section; add it with pointers to the runbook, the implementation files, ADR 0013/0006 and the audit that motivated the decision. Replace the ls -1t rotation pipeline with a glob + sort -r (SC2012); timestamped filenames make lexical order chronological. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lands ADR 0017 (Accepted) and its implementation in one PR. Closes the runbook TODO ("Backups are the operator's responsibility... TODO document") that the reliability sprint stopped on.
docs/decisions/0017-database-backups-two-leg.md. Two legs proportionate to a ~20 MB database on a shared VPS: nightly on-boxpg_dump -Fc(7 kept) for logical-error recovery, plus a gpg-encrypted rclone push to Google Drive (30-day retention) for disk/box loss. Documents the four rejected alternatives (Storage Box: cost; R2: couples to the unfinished ADR 0013 migration; B2: new identity to manage; dumps-in-Git: security and hygiene) and the Hetzner whole-VM backups as coarse fallback only.api/deploy/backup/cercol-db-backup.sh. Root cron viarunuser -u postgrespeer auth; integrity check (pg_restore --list); exits non-zero on any failure, logs to/home/cercol/logs/db-backup.log, and maintains adb-backup.FAILEDmarker removed on next success. Deliberately breaks with theMAILTO=""silent-cron pattern.api/deploy/cron/cercol-db-backup, daily 03:15 UTC (checked against existing slots: 02:00 crawl, 03:00 Sun Bing, 04:00 purge). Ships in the repo; installed manually by the operator, same pattern as the SEO ingest crons. Nothing is installed on the server by this PR.shell-lintjob (shellcheck overapi/deploy/backup/*.sh).Resolved design details (in the ADR before acceptance)
/root/.cercol-backup-passphrase, created by the operator, never committed.Verification
bash -nclean; shellcheck runs in CI (not available locally).api/deploy/*.🤖 Generated with Claude Code