v0.4.0 — import provenance and ingest ETA

Nothing in the UI said how old the archive was. This release answers that, and fixes two compose defects that a full teardown exposed.

Import provenance

A badge in the top bar carries the completion time of the last import, absolute and relative, linking to a new /imports page with one card per run: start, finish, duration, throughput, counts, and the oldest and newest message in the source file.

The badge reads the last cleanly completed run rather than max(finished_at). An interrupted or failed run has a finished_at too, and dating the archive from one would claim a freshness it does not have. No completed run means no badge rather than a guess.

A Takeout export carries no timestamp of its own, so newest message is the closest available stand-in for when the dump was taken.

Ingest ETA

Every checkpoint line now says when the run will finish:

checkpoint: 6000/8000 messages, 6000 new, 0 failures (562 msg/s, 7.6 MiB/s) — ETA 17:47:59 EDT (5s)

Extrapolated from bytes rather than message count — message size on a real export spans five orders of magnitude — and averaged over the whole run rather than the last batch. The batch rate beside it swung between 218 and 1,680 msg/s in testing, which is useful for spotting a stall and useless for predicting an end time. On a 277,020-message import the estimate landed four seconds from the actual finish.

Compose fixes

Both were latent defects that only surfaced on a genuine teardown-and-rebuild, a path that had never been exercised from an empty volume.

• POSTGRES_DB now tracks GMAIL_ARCHIVE_DB. It was hardcoded while every DSN read the variable, so any non-default database name produced a cluster the app could not connect to. migrate creates schemas, not databases, so the stack could not bootstrap at all — and pg_isready reports success for a database that does not exist, so Postgres went healthy and the failure surfaced later as a traceback.
• The default image tag now tracks the project version. It sat at 0.2.8 while the database had been migrated to the 0.3.0 schema, so docker compose up -d pulled code querying a dropped column and every message detail page returned 503.

Tests pin both, each verified by reintroducing the exact regression and watching the guard fail.

Upgrading

No migration. 0.3.0 and 0.4.0 share a schema.

v0.1.0 — first release

First tagged release. All ten planned build phases are complete, and the archive is in real use: 277,017 messages ingested from a twenty-year Google Takeout export and served by the web UI.

This is a personal archival tool published in the open. There is no support and no backwards-compatibility commitment. Read Known defects before trusting it with anything — some parts do not do what their own documentation says.

What it does

Takes a Google Takeout Gmail .mbox export and turns it into something you can actually use twenty years later: a searchable local archive with no cloud service in the loop, no account, and no network dependency at runtime.

Raw message bytes go to a content-addressed blob store on disk. Only derived metadata and the search index go in Postgres — so a pg_dump of a 30 GB archive stays small enough to be worth taking regularly, and the bytes are recoverable independently of the database.

Features

Ingest

• Resumable. The checkpoint lives in the database, not a sidecar file, so a container kill mid-run resumes where it left off. (See defects — this has a bug.)
• Idempotent. Re-ingesting the same file adds nothing; ON CONFLICT DO NOTHING at the row level makes even a partial batch replay safe.
• Parallel. A process pool per CPU by default; workers pread their own byte ranges, so there is no shared file handle to contend on. --workers and --batch-size tune it.
• Batched COPY into Postgres via temporary staging tables, with live msg/s and MiB/s in the checkpoint log.

Parser built for twenty-year-old mail

parse() never raises, for any byte string — a hypothesis property test holds that line. Real exports contain things that kill naive parsers, and each is handled and recorded rather than fatal. Fourteen warning codes are stored per message and shown in the UI:

header-undecodable · date-missing · date-unparseable · date-implausible · date-tz-out-of-range · message-id-missing · body-undecodable · charset-unknown · nul-stripped · surrogate-stripped · search-text-truncated · unquote-ambiguous · attachment-undecodable · structure-unparseable

Specific hazards this exists to contain: Postgres text cannot hold NUL, and decoded bodies do contain them; lone surrogates arrive via surrogateescape; the tsvector 1 MB limit; timezone offsets outside Postgres's ±15:59; and 8-bit bytes in structured headers, which make Message.get() hand back a Header object instead of a string and take out parsedate_to_datetime from below.

Search and browse

• Postgres full-text search over a GIN-indexed tsvector, with websearch_to_tsquery syntax — quoted phrases and -excluded terms — and ts_headline snippets with hit highlighting.
• Sortable: newest first (default), oldest first, or relevance.
• Keyset pagination on (internal_date desc nulls last, raw_sha256 desc), matching its index exactly, so paging deep into a few hundred thousand messages does not degrade.
• Thread view, label listing with counts, and a message detail page rendering HTML bodies through nh3 inside a sandboxed iframe under a script-src 'self' CSP.
• Raw source view in the browser, plus raw download served Content-Disposition: attachment with nosniff.

Synthetic fixture generator

Probably the most reusable piece here. A real Takeout export cannot be a test fixture in a public repository, so the project generates its own input:

gmail-archive gen-fixture /tmp/fixture.mbox --count 500 --seed 1
gmail-archive gen-fixture /tmp/menu.mbox --pathologies list

26 pathologies — quoted-from, body-nul, charset-nonexistent, deep-nesting, header-8bit, attach-path-filename, base64-bad-padding and more. With no --pathologies it emits a realistic mix at defect rates measured against a real twenty-year export; naming them guarantees each appears at least once, including several no real export contains. --seed is byte-reproducible. Every generated address is confined to an RFC 2606 reserved domain by construction, asserted against the generated bytes rather than the call sites.

You can exercise the entire project without owning an mbox.

Export and integrity

• export --format mbox|eml, filtered by --label, --query or --limit.
• verify reconciles the database against the blob store and the source-file sightings, reporting orphaned blobs, missing blobs, and count mismatches.
• verify --deep re-hashes every blob against its own filename. That is the payoff of content addressing: integrity checking needs no stored checksum, because the name is the checksum.

Operations

• Docker Compose: web container plus a pinned postgres:18, a documented postgresql.conf, health checks, and a non-root runtime user.
• Numbered .sql migrations applied by an in-repo runner — each migration and the row recording it commit in one transaction.
• Pre-commit hooks that reject staged .mbox files, anything under blobs/, oversized files, and secrets. On a public repo holding real mail, this is the safety net.
• /healthz (liveness, never touches Postgres), /readyz (real round-trip), /version (build metadata).

Requirements

Docker with Compose. For local development, Python 3.13 and uv. Postgres 18 comes from Compose.

See the README for setup and the runbook for operations.

After a large ingest, run vacuum (analyze). Stale planner statistics make search take seconds instead of milliseconds. On a 277k-message archive it takes ~17 seconds and moves search from multi-second to under 100 ms.

Known defects

Found by a full-repo review on 2026-08-06. The recurring theme: the places with no tests are the places that do not work.

What	Effect	Issue
Ingest never unquotes mboxrd	raw_sha256, blobs and body_text all carry >From quoting, contrary to ADR-002. Fixing this changes every hash — it is a re-ingest, not a migration, so treat any archive built with 0.1.0 as needing one	#10
Resume checkpoint uses the last worker to finish, not the furthest offset	An interrupted-and-resumed ingest can skip messages silently. Prefer a full re-ingest over a resume	#12
IMAP login rejects every credential	The IMAP server is built and wired up but unusable	#11
imap-backfill renumbers UIDs by position	Cannot be re-run after a second ingest; aborts on a primary-key collision	#13
Undated messages (~2.7%) are unreachable by browsing	Stored and searchable, but the keyset walk stops before them	#15

Also worth knowing: there is no CI (#20) — the pre-commit hooks are the only gate — and nothing tests the IMAP package at all (#16).

The full list is at the end of docs/progress.md.

Not in this release

Gmail API sync exists as an interface with respx-mocked tests only — there is no live sync, by design (plan.md Phase 8). Autocomplete is #28.