Repository navigation
Design
The pipeline maintains a catalog of articles; each article carries one artifact per (stage, source); an artifact is reused when its fingerprint still matches, and recomputed otherwise.
Everything else follows from that. There is no second way to find out what is cached, and no stage reaches into the cache on its own.
Article ──< Alias (pmid / pmcid / doi / neurostore -> article)
│
└─────< Artifact (stage, source) -> status, fingerprint, blob, summary
│
└──< Attempt (append-only: when, ok, error)
An article gets an opaque article_id the first time it is seen. Every
identifier ever associated with it becomes an alias pointing at that id.
Enrichment adds aliases; it never changes the id.
cat.register(Identifier(pmcid="PMC10634720")) # -> "a7Kq2mNp"
cat.register(Identifier(pmcid="PMC10634720",
pmid="37961286")) # -> "a7Kq2mNp", + alias| assigned | shape | exists when | |
|---|---|---|---|
article_id |
by the catalog, at registration | 12 lowercase base32, lsfb2vznhrfz
|
always |
base_study_id |
by Neurostore, during upload | mixed-case shortuuid-12, 5Qk2mNpXyJKH
|
only after a successful upload |
article_id has to exist before anything is downloaded, because download,
extract, metadata and analyses all key on it. base_study_id cannot exist until
upload has run — where it is either created or discovered by matching an
existing base study on DOI or PMID — and most articles never get one. So they
are separate, and the catalog records the mapping: on a successful upload the
base_study_id is written as a neurostore alias, which makes
ingest show <base_study_id> work.
The two shapes are deliberately different so they cannot be confused where both appear.
One string: "<kind>:<value>", from the strongest identifier the article had
when it was first registered — pmcid, else pmid, else doi, else neurostore.
Nothing else contributes.
| identifiers held at first registration | seed | id |
|---|---|---|
| pmcid, pmid, doi | pmcid:PMC10634720 |
lsfb2vznhrfz |
| pmid, doi | pmid:37961286 |
ckw3v7eecmlk |
| doi only | doi:10.1016/j.neuroimage.2023.120 |
6rgtrdiydfza |
The id never changes once assigned. A doi-seeded article that later learns
its pmcid keeps 6rgtrdiydfza and gains an alias; its artifacts are untouched.
That is the invariant the rest of the catalog rests on.
The consequence is that the id is reproducible for a given first-observation,
not canonical: a catalog that happened to see the pmcid first would have named
the same article lsfb2vznhrfz. Two catalogs agree only when they saw the same
identifier first — true for a repeated migration over the same caches, which is
what the reproducibility is for, and false in general. Nothing depends on it
being canonical: article_id never leaves the catalog, ns-pond paths use the
Neurostore base_study_id, and a random id passes every test here except
cross-catalog agreement (test_a_random_id_would_also_be_correct).
When a merge does happen, the older article keeps its id, since it has had longer to accumulate artifacts and is therefore the smaller rewrite.
This is the fix for finding 1. Resolution is a single
indexed lookup on aliases, so identifier_aliases, expand_target_aliases
and the four _hydrate_*_from_cache copies all disappear.
When two ids that were thought distinct turn out to name one article, the
catalog records a merge (articles.merged_into) rather than deleting a row.
The older article keeps its id, since it has had longer to accumulate artifacts,
and the other's artifacts are handed over to it. Where both sides hold the same
(stage, source), the better of the two is kept — a success over a failure,
then the more recent. Nothing on disk is ever unlinked by a merge — see
Data safety.
ingest status reports any artifact left attached to a retired article, and
ingest repair hands them over.
(article_id, stage, source) -> status, fingerprint, blob_sha, summary, error
status is one of:
| status | meaning | retried? |
|---|---|---|
ok |
produced, reusable | only if fingerprint changed |
failed |
transient failure | yes, under backoff |
permanent |
404, no PMC record, paywalled | never, without --refresh
|
skipped |
source cannot handle this id | no |
Recording failed and permanent is the fix for
finding 7: a paywalled article stops being retried on
every run, and tmp/failed_pubget_downloads.txt stops being necessary.
ingest add --search records each run in two tables beside the aliases:
searches (the query as given, the years, the time, and per provider the query it
was sent and its record, article and no-ID counts) and search_hits (provider and
the article a hit resolved to, after registration and enrichment). Hits are read
through articles.merged_into, so a merge needs no rewrite and any process may
merge. ingest searches manifest selects a run's articles by provider; see
#119.
Today: force_redownload, force_reextract, ignore_cache_stages,
use_cached_inputs, cache_only_mode. Their interactions are undocumented and
partly contradictory.
Instead, each stage declares what its output depends on:
def fingerprint(self, ctx, article):
return fp("extract", self.source, EXTRACTOR_VERSION,
upstream=ctx.artifact(article, "download", self.source).fingerprint)An artifact is fresh iff status == "ok", its fingerprint equals the one
computed now, and its blob still exists on disk. Otherwise it is work. Change
the coordinate prompt and every analysis goes stale automatically — which is
what create_analyses._stamp_matches already does correctly for one stage, and
is here generalised to all of them.
The only override is --refresh <stage>, which ignores freshness for that stage.
The catalog row holds identity, status, fingerprint and a small summary (the handful of fields that get queried or displayed). Anything large is gzipped into a content-addressed store and referenced by hash:
<cache_root>/blobs/ab/abcd1234….json.gz
Identical payloads deduplicate for free. For metadata this is the fix for
finding 4: raw_metadata goes to a blob, the row
keeps title/authors/abstract/journal/year/license/oa, and the 5.2 GB index
becomes a ~600 MB one.
Catalog is a context manager holding a single WAL connection with the schema
created once at open. load_index()-per-lookup, and the file descriptor it
leaked each time, are gone (finding 5).
A stage is four things and nothing more:
class Stage(Protocol):
name: str
requires: str | None # upstream stage, or None
def sources(self, ctx) -> Sequence[str]
def fingerprint(self, ctx, article, source) -> str
def execute(self, ctx, batch: list[Work]) -> Iterator[Outcome]The scheduler owns everything else: resolving the selection, asking the catalog
which articles are not fresh, batching them, calling execute, recording
outcomes and attempts. Stages never touch the catalog directly, so there is
exactly one place where "is this cached?" is decided.
download tries sources in the configured download_sources order until one
succeeds, so an article can end up with several successful downloads — 39,533
of 612,685 carry two, and 15 carry three.
extract does not fan out over them. The analyses stage consumes a single
extraction, so extracting the others produces results nothing reads. It queues
at most one unit of work per article, picking the highest-priority source that
has not been ruled out, and only moves down the list when a source has failed
and run out of retries. A success from any source makes the article fresh.
A source that has since been dropped from download_sources — a legacy
migration can record one — sorts after the configured ones rather than
becoming unreachable.
download → extract → [reflist] → [references] → [passages] → metadata → triage → analyses → [prose] → [resolve] → space → upload → sync
Each stage requires the one before it, except that triage needs both the
tables extract keeps and the abstract metadata fetched. triage decides
which tables earn a model call, and analyses makes the call. space fills
in the coordinate space for the tables analyses left OTHER, reading it from
the table's caption and the article's Methods and Results
(#51). Its payload has the same shape as analyses,
so upload and sync read space and never analyses directly.
With prose_model set, prose has the same two steps as tables
(#54). passages, its extraction, reads the
article's own download — its Methods and Results and figure legends, the whole
text when no section is recognised — whether or not an extraction succeeded,
and keeps the passages holding coordinates. metadata runs after both
extractions and before both models, and also reaches an article whose only
coordinates are in its passages. prose, the model call, reads each passage
with the article's title and abstract, and reads again when the metadata
changes. resolve merges its results with the tables' analyses, when there
are any, into the same payload plus a prose collection; space then reads
resolve instead of analyses. Where the prose adds nothing, resolve names
the analyses artifact it passed through and space keys its fingerprint on
that, so switching prose on leaves processed articles fresh.
reflist and references run only when named (--stage), since nothing
downstream reads them yet (#58,
#60). references keeps, for each extraction, the
reference list and every in-text citation with offsets for the marker and its
sentence in that extraction's text:
- XML links are exact (
xref,ce:cross-ref); - publisher HTML links are placed by their context;
- where the download marks nothing, the text is matched against
reflist's Crossref list.
reflist fetches that list by DOI, and OpenAlex names its bare-DOI entries.
download and extract remain thin adapters over the four existing
extractors — the heterogeneity of pubget / elsevier / ace / pdf is the reason
the system exists, not an accident to be normalised away.
plan yields batches. execute consumes a batch. Results are recorded per
batch. A run over a fifty-article manifest touches fifty articles' worth of
memory, which is the fix for finding 6.
The old CLI had run, search, download, extract, create-analyses,
upload-analyses, index-legacy-downloads, and a sync that was pass. Some
took --manifest, some took a positional JSON file, and the extract command
hydrated its inputs differently from the orchestrator.
The new one has five verbs and one mental model:
ingest add <ids | --query | --file> register articles
ingest run [--stage …] [--select …] advance them
ingest status [--stage …] what state is everything in
ingest show <id> everything known about one article
ingest migrate <old-cache-root> import pre-refactor caches
ingest refresh-text [--dry-run] rebuild extraction text in place
refresh-text is for a change that touches only the text, such as pubget keeping
superscripts. It rebuilds each JATS extraction's text from its download and keeps
the extraction's fingerprint, so no stage re-runs. It replaces the corpus copy of a
text only where that copy is byte-identical to the old text
(#59).
ingest add 37961286 PMC10634720 10.1016/j.neuroimage.2023.120 # ids, any kind
ingest add --file pmids.txt # one per line
ingest add --manifest staging/manifests/fmri-pet-2010.jsonl # existing JSONL
ingest add --query '(fmri OR PET) AND ...' --start-year 2010 # PubMed searchadd resolves and registers; it downloads nothing. It prints what was new and
what was already known, so adding a manifest twice is visibly a no-op. A
--search run is kept and given an id; ingest searches list and
ingest searches manifest <id> --found-by/--only-found-by <provider> read it back.
The pipeline is otherwise one-way: sync writes ns-pond, and nothing reads
Neurostore looking for articles. --from-neurostore is the inbound path. It
reads the database the upload stage writes to, so the two cannot drift, and
takes group-level base studies that are in a studyset and have no
source='llm' study:
- in a studyset, because that is what marks a base study as one someone wants;
- no llm study, because that is exactly what this pipeline would add;
- group level, because meta-analyses are not papers to extract coordinates from.
It registers the base_study_id alongside the DOI, PMID and PMCID from the same
row. That matters: an article known only by its base_study_id is inert — no
download source can address it — so the bibliographic ids are what make it
actionable. Base studies with none of the three are skipped and counted.
Run against the staging database, the filter picks out:
base_studies, all levels : 42,930
level = 'group' : 42,751
... and in a studyset : 41,999
... and already has an llm study : 27,126
... and has NO llm study <-- work: 14,873
of those, no doi/pmid/pmcid : 218 (skipped)
actionable : 14,655
The 27,126 with an llm study line up with the 27,411 rows in the legacy upload cache, which is a useful check that the filter means what it says.
Discovery composes with migration rather than duplicating it. Migrating the legacy caches first and then discovering 295 base studies:
$ ingest migrate <legacy> -> catalog articles: 0 -> 96,418
$ ingest add --from-neurostore --limit 300
neurostore: 295 base studies with no llm study yet
295 identifiers resolved to 65 new articles (230 already known)
230 of the 295 were already in the catalog from migrated downloads, so they
gained a neurostore alias on their existing article instead of becoming
duplicates, and they plan as fresh at download — the work already done is not
redone.
ingest run # every stage, everything pending
ingest run --stage download --stage extract # just these
ingest run --select new # only never-attempted articles
ingest run --select failed # retry transient failures
ingest run --select stale # fingerprint changed
ingest run --manifest x.jsonl # restrict to a set
ingest run --refresh create_analyses # ignore freshness for one stage
ingest run --dry-run # print the plan, touch nothingDefault --select pending means new + stale + retryable-failed: "do the work
that needs doing". This is the piecemeal interface — any stage, any subset, any
time, with the cache deciding what is actually recomputed.
--dry-run prints the plan instead of executing it, so the answer to "what will
this run do?" never requires reading the code:
$ ingest run --manifest vbm-1995.jsonl --dry-run
selection: 4,812 articles from vbm-1995.jsonl
download 1,204 pending 3,608 fresh
extract 887 pending 3,201 fresh 724 blocked (no download)
create_analyses 887 pending 2,918 fresh 1,007 skipped (no tables)
upload 1,113 pending 3,699 fresh
sync 1,113 pending 3,699 fresh
$ ingest status
catalog: 463,584 articles
stage ok failed permanent skipped ready
download 463,584 2,110 18,442 0 0
extract 226,523 1,004 5,118 0 231,053
…
ready = the upstream stage succeeded and this stage has no entry yet.
Columns count articles, not artifacts, so they stay comparable with the
catalog total; a stage that runs per source takes the best state any of its
sources reached. ready answers "how much work is queued at this stage right
now", which a run's plan does not — a plan cascades, so a downstream stage
reports blocked because its upstream has not run in that pass.
$ ingest show PMC10634720 # or 5Qk2mNpXyJKH, once uploaded
article lsfb2vznhrfz
aliases pmcid PMC10634720 · pmid 37961286 · doi 10.1101/2023.10.21.563317
· neurostore 5Qk2mNpXyJKH
download pubget ok 2026-04-29 blob 3f2a… 2 files
extract pubget ok 2026-04-29 blob 91bc… 6 tables, 4 with coordinates
analyses — ok 2026-04-29 blob c40d… 4 collections, 61 coordinates
upload — ok 2026-04-29 base_study 5Qk2mNpXyJKH
Switching between staging and production used to mean editing six coupled
settings — ssh host, ssh user, container name, docker network, forward port,
password — with nothing checking the combination. main shipped a mismatched
pair as its defaults: upload_ssh_host: neurostore.xyz (staging) alongside
store-store-pgsql17-1 and nginx-proxy, which is what an unprefixed store
compose project on the shared network looks like. That is why the shipped
config did not work anywhere.
It is now one line:
neurostore_env: staging # or dev, or production| ssh | container | port | |
|---|---|---|---|
staging |
jdkent@neurostore.xyz |
neurostore-staging-store-store-pgsql17-1 |
6543 |
dev |
jdkent@neurostore.xyz |
neurostore-dev-store-store-pgsql17-1 |
6544 |
production |
james@neurostore.org |
store-store-pgsql17-1 |
6545 |
Docker names containers <compose-project>-<service>-<index>, so these are
deployment facts that cannot be derived from the hostname — hence a table
rather than a rule. The forward ports differ so two environments can be open at
once without colliding on localhost.
The profile sits above the field defaults and below everything an operator sets, so a single value can still be overridden:
neurostore_env: production
upload_local_forward_port: 7000 # this winsPrecedence, highest first: CLI flags > YAML > environment variables > environment profile > field defaults.
staging and dev were read off docker ps on the live host. production
is inferred from store/docker-compose.yml and has not been verified against
the live host — confirm it before uploading there.
Every connection logs which deployment it reached, to the console as well as the file:
[INFO] Connecting to the staging database: jdkent@neurostore.xyz
-> neurostore-staging-store-store-pgsql17-1:5432
Settings loses the dead fields (neurostore_*, verbose which nothing read)
and the five cache flags, and gains nothing. Logging gains a real level so that
an 8.1 GB pipeline.log of filelock debug output cannot happen again.
- The four extractors, and their differing storage layouts. Normalising them is a separate change with its own migration risk; the catalog records wherever the bytes already are.
- The
ns-pondoutput layout,stage1/analyses.jsonandpmids.tsv. These are a contract with pondie. - The upload SQL and its Neurostore schema assumptions.