fix(ops): backup_db.sh survives stale rename — keeps backing up real data - #59
Draft
Sbussiso wants to merge 1 commit into
Draft
fix(ops): backup_db.sh survives stale rename — keeps backing up real data#59Sbussiso wants to merge 1 commit into
Sbussiso wants to merge 1 commit into
Conversation
The opensentry → sentinel rename (PR #53) updated DATABASE_URL and this script's default DB_PATH to /data/sentinel.db, but the data file on the existing Fly volume was never renamed — it still lives at /data/opensentry.db. The backup workflow's `[ -f "$DB_PATH" ]` pre-check then hard-failed the daily backup job (run #28738858714, 2026-07-05), taking the off-platform safety net dark while production ran against the orphaned old file. Now, when the expected DB_PATH is missing, the script looks for the pre-rename fallback (opensentry.db) in the same directory and backs THAT up, emitting a prominent warning that surfaces the drift (DATABASE_URL points at sentinel.db but real data is at opensentry.db → the app is running against the wrong file). The backup proceeds on the real data instead of dying. The proper fix is still to rename the file on the volume (ops runbook), but the backup must not wait for it. Behaviour: - sentinel.db present → unchanged (no warning, no fallback) - sentinel.db missing + opensentry.db present → fallback + loud warning - both missing → die with a clearer message naming the fallback Tested all three paths locally with sqlite3 fixtures; normal path emits zero warnings (no regression).
Sbussiso
referenced
this pull request
Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
Scheduled DB Backup workflow run #28738858714 (2026-07-05) failed:
The three prior daily runs (07-02, 07-03, 07-04) all succeeded.
Root cause (verified live on the Fly machine)
The
opensentry → sentinelrename in #53 updated config but not the data file on the volume:fly.toml/ deployed machine env:DATABASE_URL = sqlite:////data/sentinel.dbbackup_db.shdefault:DB_PATH=/data/sentinel.dbsentinel-commandmachine's volume (sentinel_data, vol_4oj06ney8zqq1j9r), the file is still named/data/opensentry.db(364 KB, dated Jul 4 23:49)./data/sentinel.dbdoes not exist.Confirmed via
flyctl ssh console:So the previous successful run (07-04) targeted the old app
opensentry-commandand backed upopensentry.db. Today's run is the first cron tick against the renamedsentinel-commandapp, and the rename left the data file under its old name. The[ -f "$DB_PATH" ]pre-check then hard-failed.Why this is bigger than a backup-script bug
/api/healthreturnshealthywithout touching the DB, so the app's health check is passing even thoughDATABASE_URLpoints at a non-existent/data/sentinel.db. That means the running production app is disconnected from its real data (months of cameras/incidents/motion_events inopensentry.db) — it's either writing to a brand-new emptysentinel.dbor to nothing. The backup workflow is just the first thing to fail loudly.This PR (the safe, in-repo part)
Hardens
backend/scripts/backup_db.shso the daily backup safety net keeps protecting the real data through the rename dust instead of going dark:sentinel.dbpresent → unchanged (no warning, no fallback)sentinel.dbmissing +opensentry.dbpresent → backs upopensentry.dbwith a loud warning that surfaces the drift (DATABASE_URL ≠ actual data file → app running against the wrong file)Tested all three paths locally with sqlite3 fixtures; the normal path emits zero warnings (no regression).
bash -nclean.What this PR does NOT do (needs an ops action + your call)
The real production fix is to rename the file on the volume so the app reconnects to its data:
That's an irreversible production infra action on live data, so I'm NOT doing it from a headless webhook run — flagging it here and in a commit comment for you to execute or delegate. There's also an orphaned
opensentry_datavolume (vol_rnz5199ok6oqe3pr, no VM attached) worth cleaning up.A Linear issue is filed for the data-disconnect (source of truth for bugs).
Drafted because the prod ops fix above is the actual resolution; this PR is the defensive layer that prevents the backup from going dark in the meantime and from silently masking a future rename mishap.