Pipeline Overview

Note

For quick reproduction, only Stage 3 is required. Check Reproduction for detailed instructions.

Pipeline Overview

The benchmark runs as four sequential stages, executed in parallel tracks for the three dataset tiers — mock D6331, simulated (PBSIM3 + MIMIC), and DYN clinical cohort. The mock and simulated tiers share Stage-3 Ensemble (analysis_prep.py); DYN uses a parallel Stage-3 script (dyn_prep.py) that targets the same downstream notebook conventions. This page explains what each stage does, its key flags, and where it writes — the mechanics behind the per-figure recipes in Reproduction.

                 reads (fastq)
                      │
   ┌──────────────────▼──────────────────────┐
   │ 1. process.sh / process_sr.sh            │  run the 6 tools
   │    → reports/<db>-reports/…              │
   └──────────────────┬──────────────────────┘
                      │
   ┌──────────────────▼──────────────────────┐
   │ 2. postprocess.sh                        │  convert raw outputs →
   │    → kreport / .tre / .sylphmpa          │  notebook-readable reports
   └──────────────────┬──────────────────────┘
                      │
   ┌──────────────────▼──────────────────────┐
   │ 3. analysis_prep.py  (mock + simulated)  │  parse → per-(sample,tool,db) CSVs
   │    dyn_prep.py       (DYN cohort)        │  + ground-truth tables / cohort cache
   │    → results/metadata/<ts>/{analysis,    │
   │       dyn}-prep/                         │
   └──────────────────┬──────────────────────┘
                      │
   ┌──────────────────▼──────────────────────┐
   │ 4. figure notebooks                      │  figures + tables
   │    → results/<ts>/…                      │
   └─────────────────────────────────────────┘

The whole pipeline is keyed by a single --db {unified|default} axis: every stage runs once per database so unified and default outputs never collide.

Stage 1 — Run the tools

scripts/process/process.sh drives all six tools on long reads (PacBio + ONT) for all three tiers; process_sr.sh is the paired-end Illumina counterpart, used to profile the DYN Illumina baseline that anchors Fig 10's alpha-diversity comparison.

scripts/process/process.sh --db unified --tools all --threads 50 \
    --dataset-list data/datasets_mock_pacbio.txt

Key flags:

Flag	Meaning
--db {unified|default}	which reference to profile against
--tools <list>	comma-separated tool list, or all
--dataset <fastq>	a single input (repeatable)
--dataset-list <file>	one fastq path per line (#-comments + blanks skipped); combines with --dataset
--out <dir>	output root (default reports/<db>-reports)
--db-dir <dir>	parent of the per-DB index trees (default data/ref_db)
--threads N	per-tool threads (default 50)
--samples KEY1,KEY2	(process_sr.sh only) filter paired samples, or all

Datasets are required. Each fastq path must look like .../<data_group>/<technology>/<sample>.fastq so the script can carry <data_group>/<technology> into the output layout. --db-dir points at the parent of the index dirs — the script appends refseq03032025/ for unified and default_db/ for default.

Stage 2 — Post-process

scripts/process/postprocess.sh wraps the three converters in sequence with shared flags. Use the wrapper unless you need just one stage.

bash scripts/process/postprocess.sh --db unified --jobs 4

Sub-script	Converts
generate_kreport.sh	Centrifuge / Centrifuger / Sourmash → kreport
ganon_report.sh	Ganon2 .rep → _reads.tre
sylph-tax.sh	Sylph profile → .sylphmpa (taxonomy mapping)

Shared flags: --db, --report (tree to scan, default reports/<db>-reports), --db-dir, --techs, --data-groups, --jobs. Derived files are written next to their inputs inside --report — there is no separate output dir. The wrapper continues past per-step failures so a missing binary in one sub-script doesn't block the others. Each sub-script is also usable standalone.

--data-groups accepts a comma-separated list of top-level subdirs to scan (e.g. ZymoMockD6331,simulated). It defaults to ZymoMockD6331,simulated — DYN is opt-in; add it explicitly when post-processing the clinical cohort.

The tool driver fires this post-processing automatically; the standalone wrapper is for re-running it.

Stage 3 — Ensemble

This stage has two scripts that share the same parser layer (scripts/analysis/utils/parser.py) but target different dataset families:

• analysis_prep.py — mock + simulated datasets (ZymoMockD6331, simulated). Builds per-(sample, tool, db) CSVs and per-dataset ground-truth tables. Run before any of the mock / simulated figure notebooks.
• dyn_prep.py — DYN clinical cohort. Builds the (cohort, tool, db, rank) cache used by dyn_alpha_div.ipynb and dyn_heatmap.ipynb. Run before any DYN figure.

Both write to a fresh timestamped directory each run so old outputs are never overwritten:

results/metadata/<YYYYMMDD_HHMMSS>/{analysis,dyn}-prep/

Downstream notebooks auto-discover the most recent timestamp via find_latest_analysis_prep_ts() / find_latest_dyn_prep_ts() in scripts/analysis/utils/results.py; pin a different one by setting TS = "<YYYYMMDD_HHMMSS>" in the notebook config cell.

python scripts/analysis/analysis_prep.py --jobs 4
python scripts/analysis/dyn_prep.py --threads 8     # DYN cache only

analysis_prep.py flags:

Flag	Meaning
--mode {detection|abundance|both}	which subtree(s) to build (default both)
--data-groups <csv>	comma-separated data-group subdirs to scan (default ZymoMockD6331,simulated; add DYN to include the clinical cohort here too, though dyn_prep.py is preferred for DYN)
--techs <csv>	technology subdirs to scan (default pacbio,ont)
--jobs N	parallel workers (default 1)
--unified-reports DIR / --default-reports DIR	override report roots
--db-dir DIR	reference-DB root holding the per-(tool, db) ete3 sqlite snapshots
--data-root DIR	root containing dataset subdirs + ground_truth/registry.csv (default data)
--out DIR	parent output dir; a fresh <ts>/analysis-prep/ is created inside (default results)
--skip-truth / --skip-reports	skip one half of the run
--dry-run	preview the planned (mode, tool, db, sample) rows with [ok] / [MISSING] markers and write a manifest CSV under results/metadata/dry-run/analysis-prep/; no parsing, no real outputs

dyn_prep.py flags are similar in spirit (--cohorts, --threads, --dry-run, --no-log-redirect) and write to results/metadata/<ts>/dyn-prep/tables/cohorts/.

Outputs:

results/metadata/<ts>/analysis-prep/
├── analysis_prep.log / .err
├── ground_truth/
│   └── <label>_gt.csv          # one per registry entry (zymoD6331, simulated_pacbio, simulated_ont, …)
└── preprocessed/
    ├── detection/<Tool>_<db>/<sample>.csv + totals.csv
    └── abundance/<Tool>_<db>/<sample>.csv

results/metadata/<ts>/dyn-prep/
├── dyn_prep.log / .err
└── tables/cohorts/
    └── {cohort}_{tool}_{db_mode}_{rank}.tsv

totals.csv carries the root + unclassified read counts that normalize kraken-style values. Run the relevant prep script before any figure notebook. Column semantics per tool are documented in scripts/analysis/utils/output_format_notes.txt. See Analysis Methods for how the values are scored.

Both prep scripts also run a resource-safety check during --dry-run against --jobs / --threads and the host's CPU + memory headroom — useful before kicking off a long run on a shared machine.

Stage 4 — Analysis notebooks

Each notebook reads the preprocessed metadata CSVs (never raw reports) and writes a figure or table. The full figure/table → notebook map is in Reproduction.

Code map

Directory	Contents
scripts/process/	tool drivers + post-processors (this page's stages 1–2)
scripts/analysis/	analysis_prep.py, dyn_prep.py + figure notebooks (stages 3–4)
scripts/analysis/utils/	shared parsers (parser.py) + taxid/eval helpers (util.py) + alpha_div.py + notebook helpers (results.py: find_latest_*_ts, save_csv, save_fig, tables_dir, figures_dir, host/data-group extractors)
scripts/build-db/	unified-database construction (Database Build)
scripts/read-analysis/	read stats, error-rate analysis, taxid→genome mapping

Output tree

reports/{unified,default}-reports/<data_group>/<technology>/<Tool>-results/   # stages 1–2
results/metadata/<ts>/analysis-prep/preprocessed/{detection,abundance}/…      # stage 3
results/metadata/<ts>/analysis-prep/ground_truth/…                            # stage 3
results/metadata/<ts>/dyn-prep/tables/cohorts/…                               # stage 3 (DYN)
results/metadata/dry-run/{analysis,dyn}-prep/                                 # stage 3 dry-runs
results/<ts>/{detection,abundance,dyn-alpha-div,dyn-heatmap}/…                # stage 4 (tables + figures)

<ts> is a YYYYMMDD_HHMMSS stamp; multiple runs accumulate side-by-side. Notebooks reuse the prep timestamp so a single run-id ties prep artifacts to the figures that consumed them.