Data package data-v1.16.0

Microbiome Repo — data package v1.16.0 (2026-10-08)

One catalog, two tiers, all ages. The registry tables (registry_*) cover every public human shotgun-metagenome study in ENA/SRA/DDBJ — 64,369 studies, all body sites and ages — with archive-only classification (host, UBERON-anchored body sites, life stages, assay, access) and the harvested BioSample tier (615,944 BioSamples with normalised site / age / sex / country). The catalog tables (gut_*) are the curated tier: every registry study of the human gut, all ages — 2,893 studies, 583,315 samples, 3,351,031 sample × field values — with age at collection, sex, BMI, country, health condition (controlled vocabulary), antibiotic exposure, subject and timepoint recovered from archive attributes (R1), supplementary tables (R2), paper full text (R3) and abstracts (R4); every value carries a verbatim evidence quote (≤ 12 words), a labelled source, a route and a confidence. Coverage over all gut samples: age 26%, sex 31%, country 91%, health condition 40%, antibiotic exposure 17%.

Start here. gut_studies.parquet (one row per study: classification, papers, per-field coverage, age-category and health-condition mix, curated depth) → gut_sample_metadata_wide.parquet (one row per sample: every field with __route / __confidence, age_category + basis, body_site_class) → gut_sample_determinations.parquet (the evidence behind every value). registry_studies.parquet is the registry; registry_runs_v1.16.0.parquet (every run) ships as a GitHub Release asset.

Infant extension. The 389 infant-gut studies were curated in greater depth in earlier releases (delivery mode, feeding mode, preterm status, gestational age, birth weight, maternal antibiotics, probiotics, HMO supplementation, NEC). Those rows are present in the gut_* tables (curated_source = infant_catalog, infant_scope = true reproduces the former infant catalog exactly: 72,358 samples) and the original deep tables (sample_metadata_wide, sample_determinations, study_metadata_wide, cohorts, universe_studies_all, sandpiper_*, …) remain in the package for reproducibility; the paragraphs and headline numbers of that extension are kept below under Infant extension (historical description).

Infant extension (historical description)

The registry_* tables cover every public human shotgun-metagenome study in ENA/SRA/DDBJ (all body sites, all ages; archive-only classification plus the harvested BioSample tier). All other tables belong to the first curated scope, the Infant Gut Shotgun-Metagenome Catalog — the description below is that scope's.

A curated, evidence-linked catalog of every public shotgun-metagenome study of the human infant gut (0–36 months) in the INSDC archives (ENA/SRA/DDBJ), with per-sample metadata recovered from archive attributes, supplementary tables and papers.

Headline numbers (computed from the tables in this package; see CHANGELOG v1.2 and v1.2.2). 389 included studies · 154,206 catalog samples (153,685 BioSample units + 521 run units from 6 one-BioSample-per-infant deposits; their 16 parent BioSamples are listed in parent_biosamples.parquet and counted nowhere else) · 174,022 sequencing runs · 618,898 sample × field determinations, each with a verbatim evidence quote (9,559 of them are out-of-scope ages that document adult_age_flag) · age scope: 67,598 samples with per-sample or per-subject infant evidence + 7,097 in studies triage judged all-infant = 74,695 age-scope infant samples, of which 72,358 are also gut/unknown body site = catalog_scope, the headline denominator (2,900 age-scope infant rows are body-site excluded/linked: skin, milk, oral, mother…); 14,410 samples sit in mixed-age deposits and 49,556 in studies without a per-study infant estimate (age unknown); 9,559 carry an out-of-range age (adult_flagged); 5,986 are mother/other samples · age at collection ≤ 1,100 d recovered for 53,669 of 74,695 age-scope infant samples (72%) for 52,649 of 72,358 catalog-scope samples (73%) and for 62,230 of 146,336 body-site infant-scope samples (43%; v1.2.0 said 54,105 of 146,631 before the v1.2.2 host/isolate exclusions; the v1.1 figure 53,802 was computed before the run-unit fix and the site's 54,150 included the 16 parent rows and rows later superseded) · precision vs curatedMetagenomicData (hi-res set, 3,498 gold rows; extraction_gold_eval_hires.csv): age 0.99, delivery mode 0.995, preterm status 1.00 (full set of 3,670 rows: age 0.976, delivery 0.995, preterm 1.00 — see tier_field_precision.csv, column gold_set).

Universe screened: 9,579 candidate studies (all METAGENOMIC WGS/WXS runs in ENA with a human signal, no taxon filter) → 389 included, 52 unresolvable after two full-text review rounds; recall on the 22 curatedMetagenomicData infant BioProjects: 17/22 by the automated cascade, 18/22 after the human-review pass (one study, decision_stage=session_model_review), the rest excluded with a coded reason (universe_studies_all.parquet).

Catalog membership: a study is in the catalog when it is in gut_studies.parquet (samples: gut_sample_metadata_wide.parquet). The catalog_status / triage_verdict / reason_code columns of universe_studies_all.parquet and study_metadata_wide.parquet are the historical INFANT-catalog screen (389 included) — do not filter the all-age catalog on them.

Files

File	Rows	What it is
sample_metadata_wide.parquet / .csv.gz	154,206	Start here. One row per sample: identifiers, body-site class, every metadata field with <field>__confidence and <field>__route, subject/timepoint, run accessions, cohort, links
sample_determinations.parquet	618,898	Long form: one row per sample × field with evidence_source, evidence_locator, evidence_quote (≤12 words verbatim), route, scope, confidence
study_metadata_wide.parquet / .csv	389	One row per included study (BioProject): title, counts, triage evidence, recoverability tiers, per-field coverage (cov_*), cohort, links
runs.parquet	174,022	Run → sample → study with library/instrument fields (join key into ENA/SRA)
sample_subjects.parquet	154,206	Subject and timepoint resolution per sample (subject_key, role infant/mother/other, t_index)
cohorts.csv	373	Cohort clusters (studies + papers sharing a cohort) with unique-infant estimates
study_paper_links.csv	973	Study ↔ paper links with PMID/PMCID/DOI and link method
sandpiper_flag_table.csv	20	Single source for every Sandpiper flag (field, level, site label, definition, used) — v1.2.2
sandpiper_study_panel_status.csv	389	Per-study panel scope label and n breakdown (age_scope of profiled rows) — v1.2.1
sandpiper_auditor_candidate_runs.csv	375	Runs whose organism label is a non-human host or a named microbe, with the v1.2.1 action taken — v1.2.1
review_genomic_only_samples.csv, review_named_microbe_metagenomic_samples.csv, review_amplicon_runs.csv	3158 / 70 / 39	Owner review lists (library_source GENOMIC on every run; METAGENOMIC runs with a named-microbe organism; AMPLICON-strategy runs) — v1.2.1
universe_studies_all.parquet	9,579	Every screened study with verdict, reason code and evidence
parent_biosamples.parquet	16	Parent BioSamples of the run-level rows (provenance only; excluded from every count)
study_verdict_history.parquet	32,231	Every triage-stage verdict per study (screen, rubric, Sonnet replicates, Opus adjudication, growth waves, BioSample re-judge, human-review rounds, final) with model, confidence, evidence and source artifact
value_history.parquet	45,591	Determinations that are not current: superseded, rejected, dropped by the group-statement audit, duplicates, moved parent rows, auditor findings — each with status, reason, replaced_by
sample_determinations_superseded.parquet	223	Subset of the above kept in the v1.1 layout (run-level supersessions + v1.2 age corrections)
confidence_tiers.csv	29	The engine confidence tiers: route × determiner × scope with the fixed confidence values they emit
tier_field_precision.csv	49	Empirical precision of each route × confidence tier per field against the cMD gold join (hi-res and full sets)
sample_unit_classification_by_study.csv	67	Per-study sample-unit class (A run-keyed, C technical multi-run, X cross-study BioSample)
human_review_queue.csv	52	Studies the pipeline could not decide, with the reason
field_coverage_summary.csv, study_field_coverage_matrix.csv		Coverage per field and per study × field
extraction_gold_eval_hires.csv		Precision/recall vs curatedMetagenomicData
sample_determinations_all.parquet	620,214	Current and retired determinations: sample_determinations columns + release_added, release_retired (null = current), package_added, retired_reason, retired_change_stage — the per-field value timeline — R2026.1
releases.csv	6	Release registry: one row per release id (package version, date, tags, DOI, headline counts, notes file) — R2026.1
RELEASE_NOTES_R2026.1.md		Generated diff report vs the previous package (studies/samples, coverage, verdict flips, findings, Sandpiper, gold, schema) — R2026.1
contribute_worklist.csv	439	Community-contribution worklist: one row per OPEN included/uncertain study (≥ 1 of the six fields age/delivery/feeding/preterm/antibiotics/probiotic below 0.5 coverage on catalog_scope), ranked by priority_score, with blocker_code, unlock_text and a prefilled contribution Issue URL — R2026.2
contribute_worklist_fields.csv	2,634	Study × field detail of the worklist (coverage, best recoverability tier, field-level blocker, evidence) — R2026.2
registry_studies.parquet	54,410	Registry tier: one row per ENA study of the human shotgun-metagenome universe (all body sites, all ages) — host_human, assay, body_sites/life_stages (config/vocab codes with UBERON anchors), population flags, access, evidence rows, classification_stage, infant-catalog verdict, scope_memberships (config/scope.yaml) — R2026.4
registry_universe_audit.csv	35	Registry enumeration audit: per ENA slice the archive count, rows pulled and completeness — R2026.4
REGISTRY_REPORT.md		Registry build report (universe counts per slice, host / assay / site / stage facets, scope sizes, deviations) — R2026.4
registry_biosamples.parquet	611,601	Registry sample tier (R2026.5): one row per harvested BioSample of the human_all registry studies outside the curated infant catalog — body site, life stage / age_days, sex, country, collection year normalised to the vocabularies (utility-model pass over distinct attribute pairs, docs/REGISTRY_S2_PILOT.md), raw key/value provenance, disease text unnormalised — R2026.5
registry_study_papers.parquet	6,398	Registry study × paper links from Europe PMC accession mentions and NCBI BioProject declared publications (deterministic relation classes; docs/REGISTRY_S2_PAPERS.md) — R2026.5
registry_bioproject_records.parquet	4,217	NCBI BioProject record per registry study: organisation, submitter, declared publications, dates, data types — R2026.5
registry_authors.parquet	52,110	Authors (with ORCID / affiliation where present) of ≤ 5 linked papers per registry study, Europe PMC core records — R2026.5
gut_sample_determinations.parquet	1,659,128	Curated scope gut_all (R2026.7): one row per sample × field for every human gut shotgun-metagenome sample, all ages — value, normalised value, route (R1 archive attribute / R2 supplementary table / R3 paper prose / R4 abstract), confidence, verbatim evidence quote and source; infant-catalog rows copied verbatim (src_track infant_catalog) — R2026.7
gut_sample_metadata_wide.parquet	579,252	gut_all scope, one row per sample: age category, age, sex, BMI, country, health condition, antibiotic exposure, subject, timepoint (+ route / confidence per field), infant-only fields, body-site class, infant_scope (== the infant catalog's catalog_scope filter) — R2026.7
gut_studies.parquet	2,837	gut_all scope studies: registry columns + per-field coverage, age-category / health-condition distributions, curated depth and source — R2026.7
gut_runs.parquet	721,678	Catalog runs (R2026.12): one row per sequencing run of a gut_all study — accessions, library strategy / source / layout, instrument, read and base counts, first_public, sandpiper_profiled and the catalog sample_key (run-unit samples keyed by run accession) — R2026.12
gut_sandpiper_sample_summary.parquet	339,626	Sandpiper community profiles for the whole catalog (R2026.12): one row per catalog sample with a SingleM/Sandpiper 2.0.0 profile (GTDB R232) — profiled runs, depth proxy, genus richness, Shannon, top genus, QC flags; the genus- and species-level long tables (gut_sandpiper_sample_genus / species, 185 + 199 MB) ship as the release asset gut_sandpiper_profiles.zip — R2026.12
gut_sandpiper_pca_scores.parquet	335,956	Genus-level CLR-PCA scores (pc1-pc5) of every profiled sample with root coverage >= 2 — the data behind Atlas › PCA — R2026.12
gut_sandpiper_pca_loadings.parquet	383	Genus loadings of the same PCA — R2026.12
gut_sandpiper_pca_variance.csv	5	Explained variance ratio per component — R2026.12
gut_sandpiper_study_coverage.csv	2,837	Sandpiper join coverage per catalog study — R2026.12
DATA_DICTIONARY.md		Every column, every vocabulary
getting_started.ipynb		Load, filter, join, plot

Age scope and roles (new in v1.2)

body_site_class says where a sample comes from; it says nothing about the subject's age. Use age_scope for that: infant_evidenced (age ≤ 1,100 d, preterm status or gestational age committed on this sample, or on another sample of the same resolved subject), study_all_infant (no per-sample evidence, but triage estimated ≥ 90 % of the study's infant-scope samples to be infants), age_unknown_mixed_study (triage estimated the study as mixed-age), age_unknown_no_study_estimate (no per-sample evidence and no per-study estimate), adult_flagged (a committed age > 1,100 d), non_infant_role (mother/other by attribute or sample-name evidence). role is infant only when evidenced (role_source), otherwise unknown; adult/child are set from the out-of-range age. Headline coverage is computed on age_scope ∈ {infant_evidenced, study_all_infant}; field_coverage_summary.csv carries both denominators. Mixed-age deposits (61 studies, mixed_age_deposit = True, e.g. the American Gut Project PRJEB11419 with 79 infant-evidenced rows among 7,024) contribute most of the probiotic/antibiotic/delivery values on adults — filter on age_scope before using exposure fields.

Release model (R2026.1, package 1.3.0)

This is the first numbered catalog release (R<YYYY>.<n>; VERSION.json.release_id). Every fact table
(sample_determinations, universe_studies_all, study_metadata_wide, cohorts, study_paper_links, sandpiper_*)
carries three trailing columns release_added, release_retired (null = current) and package_added; sample_metadata_wide
is derived and has none. sample_determinations_all.parquet adds the retired determinations (1,316 rows,
reconstructed from value_history) so a per-field value timeline is one ORDER BY release_added query; releases.csv is the
registry of every release. Caveat: packages before 1.3.0 had no release columns, so release_added for pre-existing rows is
reconstructed from value_history.change_stage / src_track and is exact only for rows those stages touched (adult-age
recommits, run-level rows, retired values); every other pre-1.3.0 row is labelled 1.0.0 by assumption. Full rules:
DATA_DICTIONARY.md 'Release columns' and the pipeline's docs/RELEASES.md.

How to read a value

Every value carries a route (R1 = archive sample attribute or sample-name convention (per sample); R2 = supplementary table row (per sample; 'subject_level_join' in parse_note when copied from a per-subject row); R3 = statement in the paper text applied to a defined group; R4 = abstract/ENA description statement (group; confidence ≤0.5).) and a confidence. Confidence is an ordinal reliability tier fixed by the route/engine that produced the value (confidence_tiers.csv), not a calibrated probability; its empirical precision per tier and field against the gold set is in tier_field_precision.csv (see DATA_DICTIONARY). For most analyses: use R1/R2 values at any confidence, and R3/R4 values only when confidence ≥ 0.5 — those are group-level statements applied to samples. parse_note explains derivations (unit resolution, subject propagation, date arithmetic). Rows that are no longer current (superseded, rejected, dropped by the group-statement audit) are in value_history.parquet with the reason and what replaced them; study-level verdict changes are in study_verdict_history.parquet.

Known limitations

• Age is recoverable for ~37 % of infant-scope samples; per study it is all-or-nothing. ~42k samples belong to studies whose supplementary tables are keyed by identifiers absent from the archive (e.g. TEDDY) and can only be joined with submitter help.
• antibiotic_exposure means any antibiotics before/at sampling; sources that record only current use disagree (precision 0.87 vs that definition).
• feeding_mode categories differ between sources; precision 0.89.
• Exposure fields (probiotic, HMO, NEC, birth weight, maternal antibiotics) have no external truth set — audited by blind model re-judgement only.
• 9,559 samples in included studies carry an archive age > 36 months (adult_age_flag, evidence-linked to an age_at_collection_days determination with parse_note starting out_of_scope_adult); a further 63,966 samples have no age evidence at all — see age_scope.
• Study n_samples/n_runs count the study's own archive runs; 62 BioSamples carry runs from two BioProjects and appear once in the sample table (under one study), so per-study n_samples sum to 154,268 while the sample table has 154,206 rows (n_sample_rows per study).
• Extension fields (health_condition, multiple_birth, sibling_in_study, geo_subregion) were added in a single pass and are less audited than the core fields.

Provenance

Built 2026-09-17 → 2026-09-25 with Claude-model pipelines (Haiku for screening/normalisation, Sonnet for rubric judgement and prose extraction, Opus for adjudication), deterministic parsers, and two rounds of full-text model review of the undecidable residue. Full methods: CATALOG_REPORT.md and EXTRACTION_REPORT.md in the release bundle. Licence: CC-BY-4.0 for the curated tables (archive metadata remains under INSDC terms). Citation placeholder: Infant Gut Shotgun-Metagenome Catalog data package v1.2 (release v12), 2026.

Added in v1.2.0: Sandpiper community profiles and author index

• sample_metadata_wide carries 39 Sandpiper columns (sp_*, sandpiper_url, taxonomy_db, taxonomy_version, sandpiper_version): 79,473 of 154,206 samples (51.5%) in 304 studies (sp_n_samples_profiled > 0; 302 studies have profiled runs registered under their own accession, 181 have a composition panel — see sp_panel_scope) have a SingleM profile (Sandpiper 2.0.0, GTDB R232; 87,125 of 174,022 runs). Relative abundances (*_ra) are fractions of prokaryotic coverage — see SANDPIPER_REPORT.md §3 and the DATA_DICTIONARY. Every profiled sample links to https://sandpiper.qut.edu.au/run/<run> (the RANDOM-selection run with the highest root coverage). Profiles are Sandpiper 2.0.0 outputs from Zenodo snapshot 20419175; the linked live pages may show a newer Sandpiper version. Study composition panels and sp_median_* are computed over catalog_scope rows with adequate depth only (v1.2.1).
• <field>__scope (sample / subject / group / biosample) for the 16 coverage fields, so slice exports carry the scope of R3/R4 group statements.
• study_metadata_wide adds sp_n_samples_profiled, sp_frac_samples_profiled, sp_n_runs_profiled, sp_frac_runs_profiled, sp_coverage_class, sp_frac_runs_flagged (runs whose organism/library labels fire Sandpiper's non-human-host / named-microbe / synthetic / RNA rules), sp_frac_low_complexity_profiled, sp_frac_low_depth_profiled, sp_median_spf, sp_median_known_species_fraction, sp_median_bifidobacterium_ra, sp_median_shannon_genus, sp_auditor_attention, sp_auditor_reason, and the author columns first_author, last_author, n_authors, n_author_papers, organisations.
• On-site Sandpiper tables: sandpiper_sample_summary.parquet, sandpiper_top_genera.parquet (top-15 genera + unassigned bin per sample), sandpiper_study_panels.parquet (per-study mean top phyla/genera), sandpiper_run_qc.parquet (all runs, QC flags, miss reasons), sandpiper_study_coverage.csv, sandpiper_study_qc_flags.csv, sandpiper_flag_definitions.json. The full run × taxon profiles (sandpiper_profiles.parquet, ~100 MB) are a Release asset, not in this package.
• Author index: authors.parquet (28,367 study × author rows, 10,965 distinct author keys), study_authors_summary.csv (first/last author, organisations for all screened studies), authors_index.json (search index used by the website). Names are string-matched from Europe PMC author strings and NCBI BioProject records; 103 included studies have no author on record (see AUTHORS_REPORT.md).
• VERSION.json: package version, release tag, build date and sha256 + row count of every table.