Backups: twice-daily backup of HAPI state (handoff to server-setup)

[heavygee]

Problem

HAPI has accumulated weeks of session state (~78 sessions, the entire dogfood/peer-handoff history, operator manifest, voice config). A single bad migration or disk failure wipes all of it. Currently no backups.

Proposal

Handoff: server-setup agent at session 8b052d61 owns backup infrastructure. This issue specifies what to back up and the cadence.

What needs backing up (operator-only knowledge):

Path	Purpose	Frequency	Sensitivity
~/.hapi/hapi.db	Primary SQLite store (sessions, messages, runners, fcm_devices)	twice daily	high (private messages)
~/.hapi/hapi.db-wal + -shm	WAL companions -- need them consistent	with hapi.db	high
~/.hapi/hub.env	API keys (DASHSCOPE, GEMINI, ELEVENLABS, JWT secret, access token)	twice daily	secrets
~/.config/hapi/driver-manifest.yaml	Operator soup manifest (which feature layers ship in the daily)	twice daily	low
~/.hapi/driver-status.json	Driver-stack coordination state	twice daily	low
~/coding/hapi/docs/plans/*.md (untracked)	Operator-local plans and peer handoffs	twice daily	medium
~/voice-test-output/	Voice harness artefacts (WAVs + results.json)	weekly	low

Method:

• Use SQLite online backup API (.backup command or VACUUM INTO) for hapi.db -- copying the file directly while WAL is active risks corruption.
• Encrypt hub.env at rest (it contains live API keys).
• Backups should land somewhere off-box -- ideally the operator's Proxmox or a Tailscale-attached storage host.

Cadence: twice daily for now (06:00 and 18:00 local).

Acceptance

• Backup runs twice daily without operator intervention
• One verified restore drill (restore hapi.db to a scratch path, open with sqlite3, check session count matches)
• Backup destination is off the source disk
• Retention policy: keep 14 daily snapshots minimum
• hub.env backups are encrypted (e.g. age, gpg, or vault)

Handoff

• Server-setup agent: session 8b052d61
• Operator-knowledge owner: this hub operator

[heavygee]

Done — twice-daily HAPI backup + weekly restore drill live

Server-setup side. Shipped in heavygee/server-setup#5 (merged to main as 23ba35f).

Schedule

Job	Unit	Cadence
Backup	backup-hapi.timer	*-*-* 06:00:00 + *-*-* 18:00:00 local, Persistent=true
Restore drill	backup-hapi-drill.timer	Sun *-*-* 09:00:00 local

NEXT                        LEFT          UNIT                    ACTIVATES
Tue 2026-06-02 18:03:11 BST 1h 35min left backup-hapi.timer       backup-hapi.service
Sun 2026-06-07 09:13:40 BST 4 days left   backup-hapi-drill.timer backup-hapi-drill.service

Destination

• Borg repo: /mnt/drives/backup_2tb/backups/hapi.borg
• Encryption: repokey (BORG_PASSPHRASE from ~/coding/server-setup/.env). This satisfies the hub.env encrypted-at-rest requirement -- the entire repo is encrypted, so secrets (hub.env, jwt-secret.json, fcm-service-account.json, firebase-wear-companion.env) are protected at rest.
• Off-source-disk: source is /dev/nvme0n1p2 (root, where ~/.hapi lives), destination is /dev/sdb1 (/mnt/drives/backup_2tb) -- separate physical disk.

What gets captured each run

• ~/.hapi/hapi.db -- via sqlite3 .backup (WAL-safe), then PRAGMA integrity_check before borg create. Job aborts + NTFY-alerts if integrity is not ok.
• ~/.hapi/hub.env, jwt-secret.json, fcm-service-account.json, firebase-wear-companion.env, android-sdk.env, settings.json, owner-id.json, runner.state.json, voice-sessions.jsonl
• ~/.hapi/driver-status.json and ~/.hapi/operator/
• ~/.config/hapi/driver-manifest.yaml
• ~/coding/hapi/docs/plans/*.md -- untracked only (captured via git ls-files --others --exclude-standard)

WAL companions (hapi.db-wal, hapi.db-shm) aren't snapshotted separately -- SQLite's online backup API folds them in transparently.

Retention

borg prune --keep-within=15d after every successful archive. Twice-daily for 14 days = up to ~28 archives in window, with 24h margin so neither morning nor evening archive is pruned the moment it crosses the boundary. borg compact runs each cycle to reclaim space.

Restore drill log

2026-06-02 16:27:06 - Restore drill starting; archive=proxmox-hapi-20260602T162627 dir=/tmp/hapi-restore-OBsm9V
2026-06-02 16:27:07 - Extracted hapi.db at /tmp/hapi-restore-OBsm9V/.../hapi.db (111689728 bytes)
2026-06-02 16:27:07 - OK: hub.env present
2026-06-02 16:27:07 - OK: driver-status.json present
2026-06-02 16:27:07 - OK: jwt-secret.json present
2026-06-02 16:27:07 - OK: config-hapi/driver-manifest.yaml present
2026-06-02 16:27:07 - PRAGMA integrity_check: ok
2026-06-02 16:27:07 - Row counts: sessions=84 messages=73169
2026-06-02 16:27:07 - PASS: restore drill complete for archive proxmox-hapi-20260602T162627

Live DB at the time of the backup: sessions=84, messages=~73148 -- restore drill saw +21 messages, which matches the few seconds of write activity between backup start and drill.

Locations / commands

• Runbook: docs/runbooks/hapi-backups.md in heavygee/server-setup
• Backup script: scripts/backup/backup_hapi.sh
• Drill script: scripts/backup/restore_drill_hapi.sh
• Backup log: /home/heavygee/logs/backup_hapi.log
• Drill log: /home/heavygee/logs/restore_drill_hapi.log
• Prom metric: /var/lib/node_exporter/textfile_collector/borg_hapi.prom

Manual ops:

sudo systemctl start backup-hapi.service        # force a backup now
sudo systemctl start backup-hapi-drill.service  # force a restore drill now
systemctl list-timers 'backup-hapi*'            # next scheduled runs

Acceptance checklist

• Backup runs 2x/day without operator intervention
• hub.env backups encrypted at rest (borg repokey)
• One verified restore drill (logged above)
• Destination is off the source disk
• 14-day retention enforced (borg prune --keep-within=15d)

Closing as done. Re-open if anything turns up in the first week of timer runs.