v0.5.0

hotserve 0.5.0 adds off-box backups. hotserve-backup copies what each app declares in a backup block of the Caddyfile into one restic repository off the box, every hour. That covers its SQLite databases, copied safely while the app runs, and its files under shared/. Each week it proves a restore of every app. Backups sets it up. This release is the code that passed as v0.5.0-rc2 on a real box against Backblaze B2, plus #396's Deno bump to the test images and the Deno example.

Upgrading from 0.4.0

Upgrading has the steps. Three things to know:

• New dependencies: the package now needs restic, sqlite3 and ca-certificates, which apt installs with it.
• Nothing runs a backup until you set one up. The package makes a system account, hotserve-backup, and two timers, one hourly and one weekly. The timers start nothing until hotserve-backup setup has written the repository's credential file.
• A box whose apps declare no backup serves as before.

From v0.3.x, read v0.4.0's notes first: each app wants a redeploy after that upgrade.

New: off-box backups

• Declare what to keep in an app's block of the Caddyfile, with paths under the app's shared/:

  backup {
      sqlite app.db
      files  uploads
  }
  

  hotserve validates the block. With a box repository, the box also runs hotserve-backup validate on each push before applying it.

• hotserve-backup setup <url>, as root, takes an s3: URL and the storage key. It makes the repository, or finds the one already there, as on a rebuilt box. A new repository's password is shown once, to store off the box.

• After that:

  • a run every hour;
  • a drill every week, which fetches each app's newest snapshot whole and checks a 52nd of the repository's data;
  • hotserve-backup status, which any account may run;
  • hotserve-backup restore <app>, which puts an app back.

• The bucket is the protection. It is versioned, and neither of its two keys, the box's and your own machine's, can destroy a version. Nothing on the box forgets or prunes: retention runs from your own machine. So a deletion or overwrite made with either key can be undone for 180 days, if someone looks from off the box at least once every 90. A stolen login to the provider account can still destroy everything: give it a second factor.

• Run on Backblaze B2, through its S3-compatible endpoint, with all twelve release-candidate checks. An attack with the box's own key destroyed nothing. The backups were recovered into a new bucket without the account's key, and restored onto a rebuilt box under new account ids. restic's own b2: backend is refused, and setup names the S3 URL to use instead.

Behaviour changes

• Once any app declares a backup, hotserve validate and systemctl reload hotserve refuse a Caddyfile that a backup run could not read in full: for example, a declaration in a file imported from outside /etc/hotserve. A refused reload keeps the running config. hotserve run serves such a file, and says why.
• apt purge hotserve keeps /etc/hotserve-backup/repository.env, the repository's password and key, and says so. It is the only way to read the backups already made.
• A hotserve-backup account that hotserve did not make is refused by setup and by every run, until it is put right.

Not verified yet

• AWS S3 and Hetzner: the guide covers them, but nothing has run there.
• On B2, two readings are still to come: whether its lifecycle rule removes a marker hidden through the S3 endpoint, and one drill's egress on a bill.
• On a rebuilt box (#389): a storage slow to answer setup's first look gets a new password shown before it is asked for the repository's own. setup says to store the new one beside the password you keep, never in its place.

What's Changed

• feat(liveswap): an app declares what to back up by @earthlingdavey in #138
• feat(backups): the backup engine — plan, dump, upload, verify, clean by @earthlingdavey in #139
• feat(backups): restore, and the restore drill by @earthlingdavey in #140
• feat(liveswap): a backup path is 400 bytes at most by @earthlingdavey in #141
• feat(hotserve): validate and reload refuse a backup a backup run would not see by @earthlingdavey in #147
• feat(backups): status, validate, and a listing of the repository per run by @earthlingdavey in #146
• feat(backups): hotserve-backup setup, the account, and the credential file's move by @earthlingdavey in #152
• feat(backups): the units, the timers, packaging, and the docs on a fresh box by @earthlingdavey in #153
• fix(backups): the package's lifecycle, the account's rule, and an upgrade that leaves a run alone by @earthlingdavey in #155
• test(backups): e2e-backup from ~13 min to about 4 on CI by @earthlingdavey in #159
• feat(backups): the repository check, the clean-run record, and D4's backstop (PR 5) by @earthlingdavey in #161
• test(backups): cut upkeep — pin every third-party behaviour, one print format, the cache key (PR 6) by @earthlingdavey in #162
• test(backups): the pin race no longer runs out of attempts under load (PR 7, backups half) by @earthlingdavey in #166
• docs(backups): the release candidate's docs — an operator guide, a bucket no key can destroy, the checks only a real provider can answer (PR 8) by @earthlingdavey in #168
• docs(backups): keep the design record in the repository while the feature is on ice by @earthlingdavey in #179
• Merge main into backup: the box channel, v0.3.x, and the review fixes by @earthlingdavey in #321
• docs(backups): two accepted residuals from #321, as an independent trace found them by @earthlingdavey in #326
• fix(backups): status's idle line says nothing running now (#174) by @earthlingdavey in #342
• docs(backups): say what restic is, once at each door by @earthlingdavey in #343
• test(smoke): stage 5 tells a timer's scheduled run from a catch-up (#327) by @earthlingdavey in #341
• fix(backups): status's idle line is true between two units of a run by @earthlingdavey in #349
• docs(backups): the guide's intro no longer says the drill restores by @earthlingdavey in #344
• fix(backups): the drill's fetch names no cause for restic's exit 1, which it gives them all (#170) by @earthlingdavey in #346
• fix(backups): setup says what the storage answered, and what became of the password, from restic's own words (#175) by @earthlingdavey in #353
• build(package): restic, sqlite3 and ca-certificates are Depends — an image that installs no recommended package gets them too (D8, #347) by @earthlingdavey in #351
• fix(backups): setup holds a key to writing before it takes it — a read-only key is refused at setup, not a quarter of an hour into every run (RC check 6, #347) by @earthlingdavey in #352
• docs(backups): what v0.3.0-rc1 on real B2 found, in the guide's own wording (#347) by @earthlingdavey in #356
• feat(backups): Backblaze B2 through restic's S3 backend; restic's own b2: refused (#354) by @earthlingdavey in #357
• fix(backups): root's lines name root, not sudo — the guide, hotserve-backup's words, the smoke as root (#323) by @earthlingdavey in #364
• docs(backups): escape the pipes that split two residuals rows by @earthlingdavey in #367
• fix(backups): one try a request on S3 for the upload, fetch and check (#177) by @earthlingdavey in #368
• docs(backups): #177 closed — what is left of it is accepted, not owed by @earthlingdavey in #369
• Merge main into backup: resource caps, the sibling-IPC and restart fixes, the CA floor by @earthlingdavey in #371
• Record main in backup's history, after #371's squash by @earthlingdavey in #377
• build(deps): bump the go-deps group across 2 directories with 2 updates by @dependabot[bot] in #385
• Merge main (v0.4.0) into backup by @earthlingdavey in #387
• docs(hardening): modprobe's dry run can print its dependencies first by @earthlingdavey in #386
• fix(backups): setup names a bucket not there, and a password shown on an unsure look is stored beside the kept one (RC checks 1 and 7) by @earthlingdavey in #388
• docs(backups): what v0.5.0-rc1 found on B2's S3 endpoint — the twelve checks' answers, and the checks' own corrections by @earthlingdavey in #390
• docs(threat-model): backups — the asset, its principals, what each attacker gains, the residuals (#172) by @earthlingdavey in #391
• Merge main (#385, #386) into backup by @earthlingdavey in #392
• build(deps): bump Deno to 2.9.8, the example's .deno-version with the images (#384) by @earthlingdavey in #396
• fix(backups): the plan's trailing-input check reads to the end, and s… by @earthlingdavey in #397
• feat(backups): hotserve-backup — hourly off-box backups, a weekly restore drill, and restore by @earthlingdavey in #378

Full Changelog: v0.4.0...v0.5.0