-
Notifications
You must be signed in to change notification settings - Fork 0
Analysis Methods
How the profilers are scored. Two metric families measure accuracy on data with known truth - detection (did the tool find the right taxa?) and abundance (did it estimate the right proportions?) - and a set of statistical tests assess the clinical (DYN) cohort, which has no ground truth. Definitions below match the implementations in analysis_detection.ipynb, analysis_abundance.ipynb, dyn_alpha_div.ipynb, and utils/.
Presence/absence at a rank, scored against the known truth taxa.
Evaluation key. Each row is reduced to a stable identity (make_eval_key), tried in order:
-
tid:<taxid>when a positive taxid is present, -
name:<clean_name>when there's no taxid but a usable name, -
unmapped_idx:<row>as a last resort.
Predictions and truth are compared over the union of their eval keys.
Presence threshold. A taxon counts as "present" (presence_from_df) when:
- threshold
== 0: value> 0 - threshold
> 0: value>= threshold
The published detection benchmarks use a 0.001 % support threshold; the notebook sweeps others (0, 0.0001, 0.001, 0.01, 0.1) via THRESHOLD_PERCENT.
Counts and scores. Over the key union (prf_counts, prf_scores):
precision = TP / (TP + FP)
recall = TP / (TP + FN)
F1 = 2·precision·recall / (precision + recall)
Normalization to fraction-of-reads. Kraken-style tools report read counts; to compare against a percentage threshold the values are normalized using per-sample totals from totals.csv - root + unclassified reads for kraken-style tools, summed direct k-mer reads for Sourmash. This is why analysis_prep.py emits totals.csv in detection mode.
Predicted vs. truth relative abundance, aligned by taxid union and each vector normalized to sum to 1 before scoring (align_truth_and_pred).
| Metric | Definition | Reported as |
|---|---|---|
| Spearman ρ | rank correlation of predicted vs. truth abundance (scipy.stats.spearmanr) |
ρ (higher = better) |
| Bray–Curtis | `0.5 · Σ | gt − pred |
| Jensen–Shannon | JSD base-2 of the two distributions, clipped to [0, 1] | 1 − JSD (higher = better) |
Each is written to a per-rank summary CSV (*_spearman_summary.csv, *_braycurtis_summary.csv, *_jsd_summary.csv) plus the combined three-metric figure. The MIN_ABUNDANCE config controls a pre-score floor (0 keeps all taxa where either gt or pred is nonzero; >0 keeps taxa where either meets the cutoff).
- Ranks: species and genus.
-
Synonym collapsing: a synonym map (e.g.
1578 → 2742598) is applied to both predictions and truth before scoring, so synonymous taxids don't split a true match into a TP + FP pair. - Alignment: abundance metrics align on the taxid union; missing taxa get 0 mass and the best-available name.
The clinical cohort has no defined truth, so the DYN figures report diversity and cross-tool / cross-platform agreement. All operate on the cohort cache built by dyn_prep.py and the alpha-diversity values from utils/alpha_div.py.
Per sample, at a chosen MIN_ABUNDANCE threshold, each (tool, db_mode, rank) contributes one value for Shannon, Simpson, and richness (observed taxa). Compositions are thresholded and renormalized before the index is computed. These per-sample values feed the significance test below.
Two-sided MWU (scipy.stats.mannwhitneyu(a, b, alternative="two-sided")) compares diversity between two groups (e.g. a cohort vs. its Illumina baseline, or default vs. unified DB). Each sample contributes one value per (tool, db_mode, rank); the raw two-sided p-values are what Figure 8 reports, with significance encoded as *p ≤ 0.05, **p ≤ 0.01, ***p ≤ 0.001.
- Strength: nonparametric, no distributional assumptions; honest about resolution limits at small n (the exact null has a p-value floor that depends on group sizes).
- Caveat: treats groups as independent, which is only partly correct for repeated-measures cohort data — the paper acknowledges this limitation but reports the raw-p MWU as the published test.
# 1. cohort cache (one-time, ~30 min)
python scripts/analysis/dyn_prep.py
# 2. render Figure 8 (~20-30 s)
jupyter nbconvert --to notebook --execute scripts/analysis/dyn_alpha_div.ipynb- Reproduction - which notebook computes which metric
- Tools and Databases - how each tool's raw output is parsed before scoring