Skip to content

How we score

Jump to bottom
Daniil Smutin edited this page Aug 22, 2026 · 2 revisions

Scores

SamovaR scores each annotator against a known true taxid per read (ISS labels, or a true column). Look in regenerated_annotations_plots/ (tools + consensus) and reprofiled_annotations_plots/ (same plus SAMOVAR).

Per-read numbers sit on scores.png and the F1_*.png heatmaps. Profile numbers (presence and abundance of taxa, CAMI-style) sit on opal_scores.png. SamovaR always computes the OPAL-style figures in Python. The OPAL CLI is optional: it only adds HTML, .profile files, and weighted UniFrac.

Default plots remap taxIDs to genus (--rank genus). Pass --rank none for exact taxIDs. Combined CSVs on disk are not rewritten. Predictions whose taxid is not in the true set are collapsed to other (same as the heatmaps), except purity, which uses the raw predicted name. Unclassified / 0 / other are not true source taxa.

Where the files land

File Contents
scores.png Per-read bars: Accuracy, P-macro, R-macro, F1-macro, F1 (current), R² (current), TN rate, FN rate, Accuracy purity, F1 purity
opal_scores.png Profile bars: completeness, OPAL purity, OPAL F1, Jaccard, L1 (0–2), Bray–Curtis
quality_scores.csv One row per annotator; every metric below (including F1-micro, F1-weighted, TPR/FPR, OPAL TP/FP/FN)
F1_<annotator>.png Confusion heatmap; caption is current F1 plus sklearn P/R/F1 and TN/FN rates
R2_<annotator>.png Predicted vs true abundance; caption is current R²
opal/ Optional CAMI HTML + .profile files if opal.py is on PATH

Optional OPAL CLI

OPAL (Meyer et al., Genome Biology 2019) scores taxonomic profiles, not per-read labels.

./install.sh OPAL
# or
SAMOVAR_INSTALL_OPAL=1 ./install.sh

Not required: SamovaR always computes OPAL-style numbers in Python. If opal.py is on PATH, viz also writes CAMI .profile files and runs OPAL into <plots>/opal/. Disable with SAMOVAR_OPAL=0.

OPAL metrics

From the OPAL README (one line each). Presence is of taxa, not reads. Unclassified labels are ignored. L1 and Bray–Curtis use relative abundances over classified reads.

  • Weighted UniFrac error — needs the OPAL CLI and a taxonomy tree; not computed in the Python fallback.
  • L1 norm error (range 0–2) — abundance reconstruction: Σ |ât − at|.
  • TP / FP / FN of taxa — a taxon is TP if it is present in both gold and prediction.
  • Completeness = recall of taxa = TP / (TP + FN).
  • Purity = precision of taxa = TP / (TP + FP).
  • F1 of completeness and purity.
  • Jaccard index = TP / (TP + FP + FN).
  • Shannon diversity / equitability — OPAL HTML only (not in SamovaR's Python table).
  • Bray–Curtis distance — Σ |ât − at| / Σ (ât + at).

opal_scores.png plots completeness, OPAL purity, OPAL F1, Jaccard, L1 (0–2), and Bray–Curtis. CSV columns: completeness, opal_purity, opal_f1, jaccard, l1_norm, bray_curtis, plus counts opal_tp, opal_fp, opal_fn.

Per-read metrics

Let nt,c be reads with true taxid t and predicted c, N = Σ nt,c, and TP = Σt nt,t.

F1 (current) on the heatmaps and scores.png is the formula already used in viz_annotation:

P = R = TP / N    then    F1current = 2PR / (P + R) = TP / N

That is per-read accuracy (P = R by construction). A consistent wrong name (every Escherichia called Shigella) scores 0.

R² (current) is a profile-level coefficient of determination on abundances (not sklearn r2_score on labels):

R2 = 1 − Σt (n̂t − nt)2 / Σt (nt − n̄)2

nt is the true read count of taxid t; n̂t is the predicted count. 1 = counts match; 0 = no better than “every taxon has the mean abundance”; < 0 = worse than that baseline.

sklearn (accuracy_score, precision_score / recall_score / f1_score with average= and zero_division=0) on the same collapsed labels as the heatmap. Every read has one label, so F1-micro = accuracy.

Name Meaning
Accuracy Fraction of reads with pred = true. Same as F1 (current) / F1-micro.
P-macro, R-macro, F1-macro Unweighted mean of per-taxon precision, recall, F1.
F1-weighted Mean of per-taxon F1 weighted by true support (heatmap caption + CSV).
F1-micro Global TP / N; CSV only.

Macro-F1 drops when a rare taxon is missed even if overall accuracy stays high.

TN rate / FN rate (and TPR / FPR) are one-vs-rest, macro-averaged over true taxa. For each true class t: TNR = TN / (TN + FP), FNR = FN / (TP + FN). CSV stores tnr, fnr (0–1) and tn_pct, fn_pct (×100), plus tpr, fpr, tp_pct, fp_pct. Bars and F1 captions show TN rate and FN rate.

Accuracy purity / F1 purity. For each true taxid t, let m(t) be the majority classified prediction among those reads (unclassified cannot win). Accuracy purity is the read-weighted majority fraction Σ nt, m(t) / N — high even if the name is wrong, as long as it is consistent. F1 purity is that recall's harmonic mean with the precision of m(t) over all reads predicted as m(t). They match when each majority label is used for only one true taxon; they diverge when several taxa collapse onto one predicted name.

Tiny example

Six reads, true A A A B B B, pred A A A A B B (one B called A):

  • Current F1 / accuracy / F1-micro = 5/6 ≈ 0.833
  • A: P = 3/4, R = 1; B: P = 1, R = 2/3 → F1-macro ≈ 0.829
  • Accuracy purity = 5/6 (majority labels are the true labels)

If every read were called A: accuracy = 0.5, accuracy purity = 1, F1 purity = 2/3 (shared majority label hurts precision).

Implementation: src/samovar/scores.py, src/samovar/opal.py, plots in src/samovar/viz_annotation.py.

Clone this wiki locally