Skip to content

v0.7.16: Rank a document backfill by publication date, not listing order (#443)

Choose a tag to compare

@disnet disnet released this 06 Sep 18:22
· 93 commits to main since this release
6c53747
The walk and the cap eviction were selecting an author's 100 documents by
different keys. `trimAuthorDocumentsStatement` keeps the newest by
`published_at` — the key every read path orders by — while the walk took
`listed.slice(0, MAX_DOCUMENTS_PER_AUTHOR)`, the first 100 in whatever order the
PDS served. Two rules over one cap, so they fought.

Listing order is not publication order, in two shapes both live in production.
`listRecords` order is per implementation: a Bluesky-hosted PDS serves newest
rkey first, Bridgy Fed serves oldest first and rejects `reverse` with a 400. And
rkey order is *write* order, which for a publication that imported a back
catalogue in one batch spans two decades of `publishedAt` in a single page.

That alone would be cosmetic. What made it reader-visible is that the walk then
pruned every row it hadn't listed, so a reconcile deleted exactly the documents
the firehose had gotten right — one subscribed publication was a reconcile away
from losing its 2026 posts in favour of its February back catalogue, another
from showing 2007 archive imports instead of the last five years.

Three changes. `listAuthorDocuments` pages to `MAX_LIST_PAGES` instead of
stopping at the first full page: the old condition made the page bound and the
row cap the same thing, so the constant's own "up to 500 scanned" was never
true, and selecting a cap needs a set wider than the cap. It returns
`exhaustive` alongside the records, the way `AuthorCollectionListing` already
does. The walk ranks by `publishedAtMs` with a `record_uri` tie-break, matching
the eviction's `ORDER BY published_at DESC, record_uri DESC` exactly. And the
prune runs only against an exhaustive listing — a partial one falls back to cap
eviction, which ranks over everything we hold rather than over what one listing
reached, so firehose rows survive a walk that could not see them. One statement
either way, so `BACKFILL_QUERY_COST` is unchanged.

Still open: a cold-start subscribe to an oldest-first repo larger than
`MAX_LIST_PAGES` pages starts on old documents. It is no longer destructive and
converges as the author publishes, but the first render is wrong; fixing it
needs paging to the end for detected-ascending repos.

RUNBOOK §4e gains the two things this makes non-obvious: shadow-compare shares
the proxy's selection rule, so `clean` is not proof for an author at the cap,
and which drift shapes mean D1 is the better side rather than the broken one.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>