Skip to content

fix(ops): backup_db.sh survives stale rename — keeps backing up real data - #59

Draft
Sbussiso wants to merge 1 commit into
masterfrom
ci-fix/28738858714
Draft

fix(ops): backup_db.sh survives stale rename — keeps backing up real data#59
Sbussiso wants to merge 1 commit into
masterfrom
ci-fix/28738858714

Conversation

@Sbussiso

@Sbussiso Sbussiso commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

What broke

Scheduled DB Backup workflow run #28738858714 (2026-07-05) failed:

[backup_db] ERROR: database not found at /data/sentinel.db
Error: ssh shell: Process exited with status 1

The three prior daily runs (07-02, 07-03, 07-04) all succeeded.

Root cause (verified live on the Fly machine)

The opensentry → sentinel rename in #53 updated config but not the data file on the volume:

  • fly.toml / deployed machine env: DATABASE_URL = sqlite:////data/sentinel.db
  • backup_db.sh default: DB_PATH=/data/sentinel.db
  • But on the live sentinel-command machine's volume (sentinel_data, vol_4oj06ney8zqq1j9r), the file is still named /data/opensentry.db (364 KB, dated Jul 4 23:49). /data/sentinel.db does not exist.

Confirmed via flyctl ssh console:

$ ls -la /data/
-rw-r--r-- 1 root root 364544 Jul 4 23:49 opensentry.db
$ test -f /data/sentinel.db && echo YES || echo NO
NO
$ sqlite3 /data/opensentry.db ".tables"
audit_log  camera_nodes  cameras  incidents  motion_events  sentinel_config  sentinel_runs  ...

So the previous successful run (07-04) targeted the old app opensentry-command and backed up opensentry.db. Today's run is the first cron tick against the renamed sentinel-command app, and the rename left the data file under its old name. The [ -f "$DB_PATH" ] pre-check then hard-failed.

Why this is bigger than a backup-script bug

/api/health returns healthy without touching the DB, so the app's health check is passing even though DATABASE_URL points at a non-existent /data/sentinel.db. That means the running production app is disconnected from its real data (months of cameras/incidents/motion_events in opensentry.db) — it's either writing to a brand-new empty sentinel.db or to nothing. The backup workflow is just the first thing to fail loudly.

This PR (the safe, in-repo part)

Hardens backend/scripts/backup_db.sh so the daily backup safety net keeps protecting the real data through the rename dust instead of going dark:

  • sentinel.db present → unchanged (no warning, no fallback)
  • sentinel.db missing + opensentry.db present → backs up opensentry.db with a loud warning that surfaces the drift (DATABASE_URL ≠ actual data file → app running against the wrong file)
  • both missing → dies with a clearer message naming the checked fallback

Tested all three paths locally with sqlite3 fixtures; the normal path emits zero warnings (no regression). bash -n clean.

What this PR does NOT do (needs an ops action + your call)

The real production fix is to rename the file on the volume so the app reconnects to its data:

fly machine stop d89732ef243728 -a sentinel-command
flyctl ssh console -a sentinel-command -C "mv /data/opensentry.db /data/sentinel.db"
fly machine start d89732ef243728 -a sentinel-command

That's an irreversible production infra action on live data, so I'm NOT doing it from a headless webhook run — flagging it here and in a commit comment for you to execute or delegate. There's also an orphaned opensentry_data volume (vol_rnz5199ok6oqe3pr, no VM attached) worth cleaning up.

A Linear issue is filed for the data-disconnect (source of truth for bugs).

Drafted because the prod ops fix above is the actual resolution; this PR is the defensive layer that prevents the backup from going dark in the meantime and from silently masking a future rename mishap.

The opensentry → sentinel rename (PR #53) updated DATABASE_URL and this
script's default DB_PATH to /data/sentinel.db, but the data file on the
existing Fly volume was never renamed — it still lives at
/data/opensentry.db. The backup workflow's `[ -f "$DB_PATH" ]` pre-check
then hard-failed the daily backup job (run #28738858714, 2026-07-05),
taking the off-platform safety net dark while production ran against the
orphaned old file.

Now, when the expected DB_PATH is missing, the script looks for the
pre-rename fallback (opensentry.db) in the same directory and backs THAT
up, emitting a prominent warning that surfaces the drift (DATABASE_URL
points at sentinel.db but real data is at opensentry.db → the app is
running against the wrong file). The backup proceeds on the real data
instead of dying. The proper fix is still to rename the file on the
volume (ops runbook), but the backup must not wait for it.

Behaviour:
  - sentinel.db present  → unchanged (no warning, no fallback)
  - sentinel.db missing + opensentry.db present → fallback + loud warning
  - both missing          → die with a clearer message naming the fallback

Tested all three paths locally with sqlite3 fixtures; normal path emits
zero warnings (no regression).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant