PR 11 Catalog Refactor

Replacing the stage caches with one article catalog

Pull request: #11 · branch refactor/catalog-and-stages

The pipeline had no notion of article identity. Identifier.slug is recomputed from whatever identifiers are known at that moment, so enrichment silently rekeyed the cache. Three real rows from the live download/pubget index, all claiming the same PMC id:

pmc10634720||                                        (before enrichment)
37961286-10-1101-2023-10-21-563317-pmc10634720       (after enrichment)
40125571-10-1111-ejn-70081-pmc10634720               (a different article)

Every read compensated by fuzzy-matching pmid, then doi, then pmcid with LIMIT 1. Four near-copies of "hydrate from cache" grew out of that, and the CLI copy silently found fewer downloads than the others for the same manifest.

Measured against the live corpus: 10.6 GB of sqlite indices for articles whose payloads already sit on disk.

The idea

A catalog of articles; each article carries one artifact per stage; an artifact is reused while its fingerprint matches.

             ┌──────────┐
 ingest add →│ catalog  │← ingest migrate
             └────┬─────┘
                  │  ingest run
   download → extract → metadata → analyses → upload → sync

The pages

Frozen with this change:

page	what it answers
What it replaced	seven findings against the live corpus, each measured
Benchmarks	before/after numbers for the refactor
Profiling	growth curves, where the time actually goes, and what did not work

Living pages this change rewrote:

page
Design	identity, fingerprints, stages, the command line, choosing a deployment
Data safety	how migration avoids losing the corpus
Migration	command-by-command upgrade guide

Headline numbers

	before	after
single artifact lookup	736 µs	18.2 µs
memory, 40k-article run	~1.4 GB	1 MB
metadata index, queryable part (5k slice)	169.6 MB	3.5 MB
`ingest --help`	3.38 s	0.46 s
extraction pool, 5 articles	4.96 s	0.095 s

The largest win is not in any hot loop: 1.5 hours in one log fragment was spent on downloads that failed and, because only successes were cached, were retried on every subsequent run. Recording failures is worth 134–182 minutes across four runs on elsevier downloads alone.

Questions this work answered

Each of these came up in review and changed the code:

Why a UUID namespace at all? It was not earning its ceremony — see Design. Article ids are now a truncated blake2b of one string.
Does article_id change when a pmcid is added to a doi-only record? No. The alias table short-circuits before the id is computed. Chasing it found a real code/docstring mismatch in the merge rule.
Is the catalog id the Neurostore id? No, and it cannot be — one exists at registration, the other only after a successful upload.
Does sync pick up studies added through the Neurostore interface? It did not. ingest add --from-neurostore is the inbound path.
Is it easy to configure staging versus production? It was not — six coupled settings, and main shipped a mismatched pair. Now one line.