Skip to content

v0.4.0 — import provenance and ingest ETA

Latest

Choose a tag to compare

@evandhoffman evandhoffman released this 10 Aug 20:09
· 3 commits to main since this release

Nothing in the UI said how old the archive was. This release answers that, and fixes two compose defects that a full teardown exposed.

Import provenance

A badge in the top bar carries the completion time of the last import, absolute and relative, linking to a new /imports page with one card per run: start, finish, duration, throughput, counts, and the oldest and newest message in the source file.

The badge reads the last cleanly completed run rather than max(finished_at). An interrupted or failed run has a finished_at too, and dating the archive from one would claim a freshness it does not have. No completed run means no badge rather than a guess.

A Takeout export carries no timestamp of its own, so newest message is the closest available stand-in for when the dump was taken.

Ingest ETA

Every checkpoint line now says when the run will finish:

checkpoint: 6000/8000 messages, 6000 new, 0 failures (562 msg/s, 7.6 MiB/s) — ETA 17:47:59 EDT (5s)

Extrapolated from bytes rather than message count — message size on a real export spans five orders of magnitude — and averaged over the whole run rather than the last batch. The batch rate beside it swung between 218 and 1,680 msg/s in testing, which is useful for spotting a stall and useless for predicting an end time. On a 277,020-message import the estimate landed four seconds from the actual finish.

Compose fixes

Both were latent defects that only surfaced on a genuine teardown-and-rebuild, a path that had never been exercised from an empty volume.

  • POSTGRES_DB now tracks GMAIL_ARCHIVE_DB. It was hardcoded while every DSN read the variable, so any non-default database name produced a cluster the app could not connect to. migrate creates schemas, not databases, so the stack could not bootstrap at all — and pg_isready reports success for a database that does not exist, so Postgres went healthy and the failure surfaced later as a traceback.
  • The default image tag now tracks the project version. It sat at 0.2.8 while the database had been migrated to the 0.3.0 schema, so docker compose up -d pulled code querying a dropped column and every message detail page returned 503.

Tests pin both, each verified by reintroducing the exact regression and watching the guard fail.

Upgrading

No migration. 0.3.0 and 0.4.0 share a schema.