Data package data-v1.9.0

Human Shotgun-Metagenome Catalog — data package v1.9.0 (2026-09-28)

Two tiers in one package. The registry_* tables cover every public human shotgun-metagenome study in ENA/SRA/DDBJ (all body sites, all ages; archive-only classification plus the harvested BioSample tier). All other tables belong to the first curated scope, the Infant Gut Shotgun-Metagenome Catalog — the description below is that scope's.

A curated, evidence-linked catalog of every public shotgun-metagenome study of the human infant gut (0–36 months) in the INSDC archives (ENA/SRA/DDBJ), with per-sample metadata recovered from archive attributes, supplementary tables and papers.

Headline numbers (computed from the tables in this package; see CHANGELOG v1.2 and v1.2.2). 389 included studies · 154,206 catalog samples (153,685 BioSample units + 521 run units from 6 one-BioSample-per-infant deposits; their 16 parent BioSamples are listed in parent_biosamples.parquet and counted nowhere else) · 174,022 sequencing runs · 618,898 sample × field determinations, each with a verbatim evidence quote (9,559 of them are out-of-scope ages that document adult_age_flag) · age scope: 67,598 samples with per-sample or per-subject infant evidence + 7,097 in studies triage judged all-infant = 74,695 age-scope infant samples, of which 72,358 are also gut/unknown body site = catalog_scope, the headline denominator (2,900 age-scope infant rows are body-site excluded/linked: skin, milk, oral, mother…); 14,410 samples sit in mixed-age deposits and 49,556 in studies without a per-study infant estimate (age unknown); 9,559 carry an out-of-range age (adult_flagged); 5,986 are mother/other samples · age at collection ≤ 1,100 d recovered for 53,669 of 74,695 age-scope infant samples (72%) for 52,649 of 72,358 catalog-scope samples (73%) and for 62,230 of 146,336 body-site infant-scope samples (43%; v1.2.0 said 54,105 of 146,631 before the v1.2.2 host/isolate exclusions; the v1.1 figure 53,802 was computed before the run-unit fix and the site's 54,150 included the 16 parent rows and rows later superseded) · precision vs curatedMetagenomicData (hi-res set, 3,498 gold rows; extraction_gold_eval_hires.csv): age 0.99, delivery mode 0.995, preterm status 1.00 (full set of 3,670 rows: age 0.976, delivery 0.995, preterm 1.00 — see tier_field_precision.csv, column gold_set).

Universe screened: 9,579 candidate studies (all METAGENOMIC WGS/WXS runs in ENA with a human signal, no taxon filter) → 389 included, 52 unresolvable after two full-text review rounds; recall on the 22 curatedMetagenomicData infant BioProjects: 17/22 by the automated cascade, 18/22 after the human-review pass (one study, decision_stage=session_model_review), the rest excluded with a coded reason (universe_studies_all.parquet).

Files

File	Rows	What it is
sample_metadata_wide.parquet / .csv.gz	154,206	Start here. One row per sample: identifiers, body-site class, every metadata field with <field>__confidence and <field>__route, subject/timepoint, run accessions, cohort, links
sample_determinations.parquet	618,898	Long form: one row per sample × field with evidence_source, evidence_locator, evidence_quote (≤12 words verbatim), route, scope, confidence
study_metadata_wide.parquet / .csv	389	One row per included study (BioProject): title, counts, triage evidence, recoverability tiers, per-field coverage (cov_*), cohort, links
runs.parquet	174,022	Run → sample → study with library/instrument fields (join key into ENA/SRA)
sample_subjects.parquet	154,206	Subject and timepoint resolution per sample (subject_key, role infant/mother/other, t_index)
cohorts.csv	373	Cohort clusters (studies + papers sharing a cohort) with unique-infant estimates
study_paper_links.csv	973	Study ↔ paper links with PMID/PMCID/DOI and link method
sandpiper_flag_table.csv	20	Single source for every Sandpiper flag (field, level, site label, definition, used) — v1.2.2
sandpiper_study_panel_status.csv	389	Per-study panel scope label and n breakdown (age_scope of profiled rows) — v1.2.1
sandpiper_auditor_candidate_runs.csv	375	Runs whose organism label is a non-human host or a named microbe, with the v1.2.1 action taken — v1.2.1
review_genomic_only_samples.csv, review_named_microbe_metagenomic_samples.csv, review_amplicon_runs.csv	3158 / 70 / 39	Owner review lists (library_source GENOMIC on every run; METAGENOMIC runs with a named-microbe organism; AMPLICON-strategy runs) — v1.2.1
universe_studies_all.parquet	9,579	Every screened study with verdict, reason code and evidence
parent_biosamples.parquet	16	Parent BioSamples of the run-level rows (provenance only; excluded from every count)
study_verdict_history.parquet	32,231	Every triage-stage verdict per study (screen, rubric, Sonnet replicates, Opus adjudication, growth waves, BioSample re-judge, human-review rounds, final) with model, confidence, evidence and source artifact
value_history.parquet	45,591	Determinations that are not current: superseded, rejected, dropped by the group-statement audit, duplicates, moved parent rows, auditor findings — each with status, reason, replaced_by
sample_determinations_superseded.parquet	223	Subset of the above kept in the v1.1 layout (run-level supersessions + v1.2 age corrections)
confidence_tiers.csv	29	The engine confidence tiers: route × determiner × scope with the fixed confidence values they emit
tier_field_precision.csv	49	Empirical precision of each route × confidence tier per field against the cMD gold join (hi-res and full sets)
sample_unit_classification_by_study.csv	67	Per-study sample-unit class (A run-keyed, C technical multi-run, X cross-study BioSample)
human_review_queue.csv	52	Studies the pipeline could not decide, with the reason
field_coverage_summary.csv, study_field_coverage_matrix.csv		Coverage per field and per study × field
extraction_gold_eval_hires.csv		Precision/recall vs curatedMetagenomicData
sample_determinations_all.parquet	620,214	Current and retired determinations: sample_determinations columns + release_added, release_retired (null = current), package_added, retired_reason, retired_change_stage — the per-field value timeline — R2026.1
releases.csv	6	Release registry: one row per release id (package version, date, tags, DOI, headline counts, notes file) — R2026.1
RELEASE_NOTES_R2026.1.md		Generated diff report vs the previous package (studies/samples, coverage, verdict flips, findings, Sandpiper, gold, schema) — R2026.1
contribute_worklist.csv	439	Community-contribution worklist: one row per OPEN included/uncertain study (≥ 1 of the six fields age/delivery/feeding/preterm/antibiotics/probiotic below 0.5 coverage on catalog_scope), ranked by priority_score, with blocker_code, unlock_text and a prefilled contribution Issue URL — R2026.2
contribute_worklist_fields.csv	2,634	Study × field detail of the worklist (coverage, best recoverability tier, field-level blocker, evidence) — R2026.2
registry_studies.parquet	54,410	Registry tier: one row per ENA study of the human shotgun-metagenome universe (all body sites, all ages) — host_human, assay, body_sites/life_stages (config/vocab codes with UBERON anchors), population flags, access, evidence rows, classification_stage, infant-catalog verdict, scope_memberships (config/scope.yaml) — R2026.4
registry_universe_audit.csv	35	Registry enumeration audit: per ENA slice the archive count, rows pulled and completeness — R2026.4
REGISTRY_REPORT.md		Registry build report (universe counts per slice, host / assay / site / stage facets, scope sizes, deviations) — R2026.4
registry_biosamples.parquet	611,601	Registry sample tier (R2026.5): one row per harvested BioSample of the human_all registry studies outside the curated infant catalog — body site, life stage / age_days, sex, country, collection year normalised to the vocabularies (utility-model pass over distinct attribute pairs, docs/REGISTRY_S2_PILOT.md), raw key/value provenance, disease text unnormalised — R2026.5
registry_study_papers.parquet	6,398	Registry study × paper links from Europe PMC accession mentions and NCBI BioProject declared publications (deterministic relation classes; docs/REGISTRY_S2_PAPERS.md) — R2026.5
registry_bioproject_records.parquet	4,217	NCBI BioProject record per registry study: organisation, submitter, declared publications, dates, data types — R2026.5
registry_authors.parquet	52,110	Authors (with ORCID / affiliation where present) of ≤ 5 linked papers per registry study, Europe PMC core records — R2026.5
gut_sample_determinations.parquet	1,659,128	Curated scope gut_all (R2026.7): one row per sample × field for every human gut shotgun-metagenome sample, all ages — value, normalised value, route (R1 archive attribute / R2 supplementary table / R3 paper prose / R4 abstract), confidence, verbatim evidence quote and source; infant-catalog rows copied verbatim (src_track infant_catalog) — R2026.7
gut_sample_metadata_wide.parquet	579,252	gut_all scope, one row per sample: age category, age, sex, BMI, country, health condition, antibiotic exposure, subject, timepoint (+ route / confidence per field), infant-only fields, body-site class, infant_scope (== the infant catalog's catalog_scope filter) — R2026.7
gut_studies.parquet	2,837	gut_all scope studies: registry columns + per-field coverage, age-category / health-condition distributions, curated depth and source — R2026.7
DATA_DICTIONARY.md		Every column, every vocabulary
getting_started.ipynb		Load, filter, join, plot

Age scope and roles (new in v1.2)

body_site_class says where a sample comes from; it says nothing about the subject's age. Use age_scope for that: infant_evidenced (age ≤ 1,100 d, preterm status or gestational age committed on this sample, or on another sample of the same resolved subject), study_all_infant (no per-sample evidence, but triage estimated ≥ 90 % of the study's infant-scope samples to be infants), age_unknown_mixed_study (triage estimated the study as mixed-age), age_unknown_no_study_estimate (no per-sample evidence and no per-study estimate), adult_flagged (a committed age > 1,100 d), non_infant_role (mother/other by attribute or sample-name evidence). role is infant only when evidenced (role_source), otherwise unknown; adult/child are set from the out-of-range age. Headline coverage is computed on age_scope ∈ {infant_evidenced, study_all_infant}; field_coverage_summary.csv carries both denominators. Mixed-age deposits (61 studies, mixed_age_deposit = True, e.g. the American Gut Project PRJEB11419 with 79 infant-evidenced rows among 7,024) contribute most of the probiotic/antibiotic/delivery values on adults — filter on age_scope before using exposure fields.

Release model (R2026.1, package 1.3.0)

This is the first numbered catalog release (R<YYYY>.<n>; VERSION.json.release_id). Every fact table
(sample_determinations, universe_studies_all, study_metadata_wide, cohorts, study_paper_links, sandpiper_*)
carries three trailing columns release_added, release_retired (null = current) and package_added; sample_metadata_wide
is derived and has none. sample_determinations_all.parquet adds the retired determinations (1,316 rows,
reconstructed from value_history) so a per-field value timeline is one ORDER BY release_added query; releases.csv is the
registry of every release. Caveat: packages before 1.3.0 had no release columns, so release_added for pre-existing rows is
reconstructed from value_history.change_stage / src_track and is exact only for rows those stages touched (adult-age
recommits, run-level rows, retired values); every other pre-1.3.0 row is labelled 1.0.0 by assumption. Full rules:
DATA_DICTIONARY.md 'Release columns' and the pipeline's docs/RELEASES.md.

How to read a value

Every value carries a route (R1 = archive sample attribute or sample-name convention (per sample); R2 = supplementary table row (per sample; 'subject_level_join' in parse_note when copied from a per-subject row); R3 = statement in the paper text applied to a defined group; R4 = abstract/ENA description statement (group; confidence ≤0.5).) and a confidence. Confidence is an ordinal reliability tier fixed by the route/engine that produced the value (confidence_tiers.csv), not a calibrated probability; its empirical precision per tier and field against the gold set is in tier_field_precision.csv (see DATA_DICTIONARY). For most analyses: use R1/R2 values at any confidence, and R3/R4 values only when confidence ≥ 0.5 — those are group-level statements applied to samples. parse_note explains derivations (unit resolution, subject propagation, date arithmetic). Rows that are no longer current (superseded, rejected, dropped by the group-statement audit) are in value_history.parquet with the reason and what replaced them; study-level verdict changes are in study_verdict_history.parquet.

Known limitations

• Age is recoverable for ~37 % of infant-scope samples; per study it is all-or-nothing. ~42k samples belong to studies whose supplementary tables are keyed by identifiers absent from the archive (e.g. TEDDY) and can only be joined with submitter help.
• antibiotic_exposure means any antibiotics before/at sampling; sources that record only current use disagree (precision 0.87 vs that definition).
• feeding_mode categories differ between sources; precision 0.89.
• Exposure fields (probiotic, HMO, NEC, birth weight, maternal antibiotics) have no external truth set — audited by blind model re-judgement only.
• 9,559 samples in included studies carry an archive age > 36 months (adult_age_flag, evidence-linked to an age_at_collection_days determination with parse_note starting out_of_scope_adult); a further 63,966 samples have no age evidence at all — see age_scope.
• Study n_samples/n_runs count the study's own archive runs; 62 BioSamples carry runs from two BioProjects and appear once in the sample table (under one study), so per-study n_samples sum to 154,268 while the sample table has 154,206 rows (n_sample_rows per study).
• Extension fields (health_condition, multiple_birth, sibling_in_study, geo_subregion) were added in a single pass and are less audited than the core fields.

Provenance

Built 2026-09-17 → 2026-09-25 with Claude-model pipelines (Haiku for screening/normalisation, Sonnet for rubric judgement and prose extraction, Opus for adjudication), deterministic parsers, and two rounds of full-text model review of the undecidable residue. Full methods: CATALOG_REPORT.md and EXTRACTION_REPORT.md in the release bundle. Licence: CC-BY-4.0 for the curated tables (archive metadata remains under INSDC terms). Citation placeholder: Infant Gut Shotgun-Metagenome Catalog data package v1.2 (release v12), 2026.

Added in v1.2.0: Sandpiper community profiles and author index

• sample_metadata_wide carries 39 Sandpiper columns (sp_*, sandpiper_url, taxonomy_db, taxonomy_version, sandpiper_version): 79,473 of 154,206 samples (51.5%) in 304 studies (sp_n_samples_profiled > 0; 302 studies have profiled runs registered under their own accession, 181 have a composition panel — see sp_panel_scope) have a SingleM profile (Sandpiper 2.0.0, GTDB R232; 87,125 of 174,022 runs). Relative abundances (*_ra) are fractions of prokaryotic coverage — see SANDPIPER_REPORT.md §3 and the DATA_DICTIONARY. Every profiled sample links to https://sandpiper.qut.edu.au/run/<run> (the RANDOM-selection run with the highest root coverage). Profiles are Sandpiper 2.0.0 outputs from Zenodo snapshot 20419175; the linked live pages may show a newer Sandpiper version. Study composition panels and sp_median_* are computed over catalog_scope rows with adequate depth only (v1.2.1).
• <field>__scope (sample / subject / group / biosample) for the 16 coverage fields, so slice exports carry the scope of R3/R4 group statements.
• study_metadata_wide adds sp_n_samples_profiled, sp_frac_samples_profiled, sp_n_runs_profiled, sp_frac_runs_profiled, sp_coverage_class, sp_frac_runs_flagged (runs whose organism/library labels fire Sandpiper's non-human-host / named-microbe / synthetic / RNA rules), sp_frac_low_complexity_profiled, sp_frac_low_depth_profiled, sp_median_spf, sp_median_known_species_fraction, sp_median_bifidobacterium_ra, sp_median_shannon_genus, sp_auditor_attention, sp_auditor_reason, and the author columns first_author, last_author, n_authors, n_author_papers, organisations.
• On-site Sandpiper tables: sandpiper_sample_summary.parquet, sandpiper_top_genera.parquet (top-15 genera + unassigned bin per sample), sandpiper_study_panels.parquet (per-study mean top phyla/genera), sandpiper_run_qc.parquet (all runs, QC flags, miss reasons), sandpiper_study_coverage.csv, sandpiper_study_qc_flags.csv, sandpiper_flag_definitions.json. The full run × taxon profiles (sandpiper_profiles.parquet, ~100 MB) are a Release asset, not in this package.
• Author index: authors.parquet (28,367 study × author rows, 10,965 distinct author keys), study_authors_summary.csv (first/last author, organisations for all screened studies), authors_index.json (search index used by the website). Names are string-matched from Europe PMC author strings and NCBI BioProject records; 103 included studies have no author on record (see AUTHORS_REPORT.md).
• VERSION.json: package version, release tag, build date and sha256 + row count of every table.