Skip to content

v1.21.0 — Scheduled off-host database backups

Choose a tag to compare

@sunbrolynk sunbrolynk released this 06 Sep 23:44
5b85909

Covers 1.21.0 in full — five merges that landed within about an hour of each
other and ship as one image, so they get one release rather than five.

Scheduled database backups, to somewhere that is not this host

The one change here that asks anything of an operator. The published compose
file gains a second service, libex-backup, running the same image as the API,
so it arrives with every deployment that pulls the new file. There is no enable
switch — configuration is the switch. With no destination set the container
starts, says it has nothing to do, and repeats that on the schedule. If you do
not want backups, do nothing: the only cost is a slightly larger image, which
now carries the PostgreSQL 16 pg_dump and pg_restore binaries.

  • Schedule — daily, weekly or monthly, at an HH:MM local to an IANA
    timezone rather than an offset, so the hour holds across a daylight saving
    change. A time that does not exist on a given day runs at the next real
    instant on that date; a time that happens twice runs once; the 31st falls
    back to the last day of a short month. A period missed while the container
    was down is taken when it returns.
  • Retention keeps eight artefacts at the defaults, not six. Six recent,
    plus the newest artefact at least thirty days old, plus the oldest one not
    yet thirty days old. The middle tier is the copy that matters when damage is
    noticed late; the third exists only so the middle one can ever hold anything,
    because six dailies delete every candidate at about six days and nothing
    would otherwise survive to reach thirty.
  • FTPS only, and explicit FTPS on port 21 — not implicit FTPS on 990, which
    is a different protocol with a similar name. Certificate verification cannot
    be turned off; a destination that cannot be verified, or one that is only
    half configured, stays inactive with the missing piece named, rather than
    uploading anyway.
  • It fails safe. Free space is checked before the dump starts. The finished
    archive's table of contents is read back before anything uploads, and an
    artefact that fails that read is kept as evidence. A destination is pruned
    only after its own successful upload, and wherever the keep set cannot be
    worked out with confidence, nothing is deleted anywhere that cycle.

.env.example gains nineteen BACKUP_* names, all with working defaults.

What this does not do. The archive is not encrypted — the transfer is
protected, the file at the far end is not, so the destination is as trusted as
the database. And nothing here restores anything; putting a backup back is
still a manual job.

One version coupling. The client tools in the image are PostgreSQL 16 and
the compose file pins a 16 server. Moving the server to 17 stops backups
outright, and a dump taken with 17 tools cannot be read back by the 16
pg_restore that would restore it. Server and client tools move together.

A blank answer from Audible no longer erases what was stored

An empty string is not a null, so the merge meant to preserve stored values
took one as a real answer: a response group carrying a field with nothing in it
overwrote the publisher, copyright, ISBN, language, cover, subtitle or title
already on a book, and an author's portrait with it. An ordinary refresh of a
book that already had a publisher could return it without one afterwards, with
nothing announcing that a field had been emptied rather than never known.

Fourteen text columns on a book now keep what is stored when what arrives is
blank, and the author's image does the same. Emptiness is measured after
trimming the full Unicode whitespace set, so a lone tab, a non-breaking space
from a copied page, or the ideographic space that is ordinary in the Japanese
catalogue all read as no answer where each used to pass as one.

What this does not cover. Zero-width characters are not whitespace in
Unicode and are not trimmed, so a value made only of those still counts as an
answer and will still replace what is stored. The book's plans list is also
left on the old merge deliberately — an empty plans array is how a book that
has left the Plus catalogue reports itself, and guarding it would pin a book in
a catalogue it no longer belongs to.

Log lines carry their context as fields

Eighteen warnings on the database read path stopped building a sentence around
the driver's error text, which is where PostgreSQL can print the contents of
the row it was working on. They now carry the operation, what identifies the
thing being read, and the failure's type, SQLSTATE and schema object as fields.
Twenty-eight lines across the seeder and the outbound Audible calls got the
same treatment for region, entity, progress and findings. Those keep the
error's own text, because an HTTP client cannot quote a stored row.

PRIVACY.md is updated alongside them, and settles a question it had left
open: a name on a background line is a row read out of Libex's own catalogue
because it was due to be refreshed, which is a different thing from the logged
name the same page calls a defect — one that arrived because somebody searched
for it.

A correction to two earlier releases

1.19.5 and 1.19.6 both said an older PostgreSQL rejects a chapter write as a
syntax error. It does not. The statement parses and fails at type resolution,
reporting that it cannot subscript a value that is not an array — so anyone
running against an external PostgreSQL 13 or earlier who searched their logs
for a syntax error found nothing and had every reason to conclude it was not
their problem. The README now says what the server actually reports. Nothing
else in those entries changes; the version requirement and the symptom stand
as published.

Carried in from before, and previously unannounced

The compose files also cap Docker's own log files — all three services across
both published files are held to five rotated 10 MB files, where json-file
keeps everything by default — and give the database container a 1 GB /dev/shm
in place of Docker's 64 MB, which is where PostgreSQL puts the shared memory a
parallel query's workers use; too small a value fails such a query outright
instead of falling back to a serial plan. DB_SHM_SIZE lowers it on a small
host. The database also gets up to two minutes to finish its shutdown
checkpoint rather than Docker's ten seconds.

This landed shortly after 1.20.0 was released, carried no version of its own,
and reached no release until now. It is called out because pulling the compose
file for the backup container above picks it up in the same action.

Also in this release

Four gaps in the test suite closed — a response-header conformance scan that
only recognised one way of setting a header, a fixture that duplicated the
settings it was meant to mirror, an unpinned /docs description, and an
assertion that a clean author walk schedules no background completion. No
behaviour changed with them.

No endpoint, parameter, response shape, field or status code changed in this
release.