Extending

Extending

How to add a new profiler, dataset, or metric to the pipeline. The design goal is that each addition touches a small, predictable set of places.

Add a new profiler

1. Execution - add a branch in scripts/process/process.sh (and process_sr.sh if it handles short reads) that runs the tool against --db-dir's index and writes into <out>/<project>/<tech>/<Tool>-results/.

2. Post-processing - if the tool's native output isn't directly parseable, add a converter (mirroring generate_kreport.sh / ganon_report.sh / sylph-tax.sh) and wire it into postprocess.sh.

3. Parser - write parse_<tool>_... in scripts/analysis/utils/parser.py returning the standardized schema (see below).

4. Registry - register the tool in analysis_prep.py _build_registries(), in both the detection and abundance dicts. Each entry needs:

   "<Tool>": {
       "parser":      <your_parse_fn>,
       "patterns":    ["*_report.tsv"],     # glob(s) for the report file
       "exclude":     [],                   # globs to skip
       "kwargs":      {},                   # extra parser kwargs
       "needs_ncbi":  True,                 # True if the parser takes an ete3 handle
       "rank_labels": {"species": "S", "genus": "G"},  # if kreport-style
       "subdir":      "<Tool>-results",     # report subdir under the tech dir
   }

5. Taxonomy (only if needs_ncbi) - if the tool emits names or non-NCBI lineages, add its ete3 snapshot path to ETE3_SUBPATHS and map each (Tool, db) pair to a snapshot in NCBI_FOR_TOOL_DB. Tools that emit native NCBI taxids set needs_ncbi: False and skip this. See Tools and Databases.

6. Default DB - document its provenance; if rebuilding the unified index, add the tool to Database Build.

Standardized parser schema

Every parser returns a DataFrame with (at least) the unified columns the downstream notebooks expect:

taxid_raw, taxid, name_raw, name, rank, value, value_type,
abundance_raw, ...tool-specific extras...

taxid/name are the cleaned values; value is the standardized numeric field that detection/abundance scoring reads. Column semantics per existing tool are catalogued in scripts/analysis/utils/output_format_notes.txt.

Add a new dataset

1. Reads - place fastqs under data/<data_group>/<technology>/<sample>.fastq (the .../<data_group>/<technology>/<sample> shape is required so the driver can derive the output layout).

2. Manifest - add data/datasets_<name>.txt, one fastq path per line (#-comments + blanks allowed). Pass it via --dataset-list.

3. Ground truth (if known) - drop the truth CSV anywhere under --data-root (default data/) and add a row to data/ground_truth/registry.csv:

   label,relpath,sep,name_col,abund_col
   <label>,<relpath-from-data-root>,",",<name-col>,<abund-col>

   Example existing rows:

   zymoD6331,ZymoMockD6331/zymoD6331_gt.csv,",",name,abundance
   simulated_pacbio,simulated/pacbio/simulated_pacbio_gt.csv,",",name,abundance

   analysis_prep.py discovers truth sources from the registry and writes a normalized <label>_gt.csv under results/metadata/<ts>/analysis-prep/ground_truth/. If the dataset has no truth (clinical-style), skip this step - it's then analyzed by cross-tool agreement only.

4. Data-group filter - if the new dataset lives under a <data_group> directory that isn't ZymoMockD6331 or simulated, either pass it explicitly via --data-groups <csv> to postprocess.sh / analysis_prep.py, or add it to the default list inside those scripts. The DYN cohort is opt-in via the same flag (default skips it because dyn_prep.py is the preferred path for DYN).

5. Notebook selection - add the dataset key to the DATASETS dict in the relevant figure notebook's config cell, with both a SAMPLE_LABEL (raw sample name used to look up the prep output on disk) and a DISPLAY_TITLE (human-readable label shown in figure titles). Select it via DATASET at the top of the notebook.

Add a new metric

Slot it into the detection or abundance notebook next to the existing metrics, and write a per-rank summary CSV - mirror the JSD-summary pattern in analysis_abundance.ipynb (compute per (tool, rank), reorder by ORDER_TOOLDB, write *_<metric>_summary.csv). See Analysis Methods for the existing set.

Conventions

• CLI flags are consistent across the shell drivers: --db, --out, --db-dir, --report, --dataset / --dataset-list, --data-groups, --techs, --jobs, --threads, --tools, --samples.
• --data-groups <csv> is the standard "which top-level project dirs to walk" filter across postprocess.sh (and its three sub-scripts), analysis_prep.py, and dyn_prep.py. Default is ZymoMockD6331,simulated; DYN is opt-in.
• Parallelism - --jobs N in the post-processors and analysis_prep.py; --threads N for the profilers and dyn_prep.py. Both prep scripts run a CPU + memory safety check during --dry-run.
• Timestamped output dirs - analysis_prep.py and dyn_prep.py each create a fresh results/metadata/<YYYYMMDD_HHMMSS>/<script>-prep/ per run; notebooks pick the latest via find_latest_*_ts() in utils/results.py. Don't read/write through hardcoded paths - go through the helpers so a new run automatically becomes the default.
• --dry-run is supported on both prep scripts and writes its manifest under the fixed results/metadata/dry-run/<script>-prep/ path (overwrites on rerun, so dry-runs never clutter the timestamp history).
• Notebook SAVE knob + helpers - every figure notebook has a SAVE = True/False switch at the top. Use save_csv / save_fig / save_if from utils/results.py for new save sites so they respect that knob and write under the consistent results/<ts>/<analysis>/... tree.
• Sample naming - analysis_prep.py preserves raw dataset prefixes in its output filenames; downstream notebooks should look up by SAMPLE_LABEL (the raw on-disk name) and only use DISPLAY_TITLE for figure-facing strings. Don't strip prefixes inside the notebook.
• ete3 is not picklable - analysis_prep.py and dyn_prep.py build one NCBIRegistry per worker; keep new parsers compatible with that (take the ncbi handle as an argument, don't hold a module-global).

Next steps

• Pipeline Overview - where each stage lives
• Analysis Methods - the scoring functions a new metric joins