Validation

Validation: choosing the polishers with data

Polishing is where assemblies silently get worse: several short-read polishers used in BGA's past (NextPolish, ntEdit, POLCA without --careful, extra Polypolish rounds) introduced errors in low-coverage regions. HashBrown therefore exposes the polishers on the command line and ships a benchmark harness so the defaults follow measurements, not habit:

• --long-polisher {medaka,none} — with modern super-accuracy basecalls, Medaka's benefit shrinks; measure it on your own data before trusting either choice.
• --short-polisher {polypolish+pypolca,polypolish,pypolca,none} — the default pair is the combination that introduced no errors in earlier tests (Polypolish and pyPOLCA --careful are both designed to leave low-confidence regions alone).

Benchmark results (2026-10, re-measured 2026-11 with HashBrown 1.1.0)

Seven tracks, every assembly scored by QUAST against a finished genome (mismatches and indels per 100 kbp, lower is better). The full per-arm tables are committed in validation/benchmarks/results/. Every track is scripted and documented in validation/benchmarks/ — data accessions, gold-standard provenance and caveats included — so each result can be reproduced from public data (the M. bovis field-strain reads are the one in-house exception).

Simulated M. bovis AF2122/97 (badread Q20 Nanopore 100x + ART Illumina 60x; exact truth): Flye alone left one indel in 4.35 Mb; every polisher combination removed it and none introduced an error. All combinations are safe at good coverage.

Real E. coli K-12 MG1655 (MicroBench R10.4.1 PromethION 100x + NovaSeq 60x; truth U00096.3): the Flye assembly was already at Q63 (2 mismatch + 6 indel positions, which are real lab-stock differences from U00096.3). Medaka (default and bacterial model) and the short-read polishers changed at most 2 positions and nothing got worse. With recent super-accuracy basecalls at 100x, polishing has almost nothing left to fix.

Real M. bovis field strain (R10.4.1 sup v4.3.0, 100x, compared against AF2122/97 — strain divergence is a constant offset, so only the differences between arms count):

Long polisher	Mismatches/100kbp	Indels/100kbp
none	19.12	5.45
medaka, matched sup model	19.10	5.22
medaka, bacterial model	19.10	4.64

Medaka never hurt, and ONT's bacterial methylation model removed 3x more indels than the matched sup model (~35 vs ~10 genome-wide). It is HashBrown's default model since this benchmark.

Non-model organisms — Listeria innocua AUSMDU00097349 (Bacillota) and Providencia rettgeri AUSMDU00026122 (Morganellaceae), real R10.4.1 sup v5.0.0 at 100x + Illumina, truth = the same isolates' Trycycler references from the Autocycler paper (Wick et al. 2025). This track exists to test whether Medaka's bacterial methylation model only helps model organisms it may have been trained on — it does not:

Sample	Arm	Mismatches/100kbp	Indels/100kbp
L. innocua	none + none	0.00	0.03
L. innocua	any polisher combination	0.00	0.00
P. rettgeri	none + none	0.04	0.11
P. rettgeri	medaka (matched sup) + none	0.02	0.02
P. rettgeri	medaka (bacterial) + none	0.02	0.02
P. rettgeri	anything + polypolish+pypolca	0.00	0.00

With modern sup v5 basecalls at 100x the raw Flye assemblies start within a handful of errors of perfect; every polisher helps or is neutral, the bacterial model equals the matched sup model on organisms far from E. coli (its clear win is on the older sup v4.3 M. bovis track above), and the defaults end both genomes error-free against the truth. Caveat: the references were themselves polished with Medaka + Polypolish + pyPOLCA, so a blind spot shared by those tools cannot be excluded. The track is fully scripted: validation/benchmarks/autocycler_nonmodel_benchmark.sh.

Real AF2122/97 PacBio RS CLR + NextSeq (truth = its own finished genome; the CLR base assembly is deliberately rough — a stress test):

Combination	Mismatches/100kbp	Indels/100kbp
none + none	55	545
none + polypolish	46	378
none + pypolca	54	538
none + polypolish→pypolca	46	377
medaka + none	111	672

Polypolish does the heavy lifting and never hurts; pyPOLCA --careful alone declines to fix a bad assembly (by design) but adds a small gain after Polypolish. Medaka on PacBio reads doubles the mismatches: never run Medaka on non-Nanopore reads (--long-polisher none).

Real MG1655 PacBio HiFi + NovaSeq (Flye --pacbio-hifi base): the HiFi assembly was near-perfect and short-read polishing made it marginally worse (+1 mismatch, +2 indels) because the Illumina reads come from a different lab stock. Polishing an already-excellent assembly with reads from a different culture moves it toward that culture.

Ion Torrent as the polishing reads — Scandinavium goeteborgense CCUG 66741 (type strain) and Vibrio sp. ED-400, each sequenced on ONT + Ion Torrent PGM + Illumina from one DNA submission (validation/benchmarks/iontorrent_polish_benchmark.sh). A deliberately rough 50x R9.4.1 Flye base was polished with nothing, with the Ion reads, or with depth-matched Illumina, against a truth built from the deep ONT + full Illumina run. Every residual indel was classified by homopolymer context:

Sample	Arm	Mism./100kbp	Indels/100kbp	Indels left	in homopolymers
S. goeteborgense	none	4.52	141.07	7,026	97%
S. goeteborgense	Ion PGM	0.19	12.36	533	92%
S. goeteborgense	Illumina	0.13	0.83	27	0%
Vibrio ED-400	none	9.43	117.76	6,808	92%
Vibrio ED-400	Ion PGM	1.15	13.65	655	98%
Vibrio ED-400	Illumina	0.99	1.18	0	—

Ion polishing removed ~91% of the indels and nearly all mismatches, and never hurt — but the errors it cannot finish are almost entirely (92–98%) homopolymer indels, the error mode it shares with Nanopore, exactly where depth-matched Illumina goes to ~zero. Hence the --ion-torrent warning: an Ion-polished Nanopore assembly still carries homopolymer indels (frameshift-sensitive analyses beware); finish with Illumina. Caveats: the truth is Illumina-polished (favoring the Illumina arm's absolute numbers — the homopolymer concentration of the Ion residue is the construction-independent readout), and the Vibrio arms share 3 base-assembly structural differences from the truth, constant across arms.

What the defaults follow from this

• Hybrid Nanopore + Illumina: keep --long-polisher medaka --short-polisher polypolish+pypolca (the default). Gains are small on good data, large on rough assemblies, and never negative on Nanopore.
• Medaka model: r1041_e82_400bps_bacterial_methylation (the default) for bacterial isolates on R10.4.1; it beat or equalled the matched sup model everywhere — including on the non-model organisms (M. bovis, L. innocua, P. rettgeri), so the gain is not model-organism overfitting.
• Recent sup basecalls at ≥100x: --long-polisher none is acceptable — the assembler is already at the error floor — but medaka costs little and still helps older basecalls (sup v4.3: −35 indels).
• Non-Nanopore long reads (PacBio): always --long-polisher none.
• Polish with reads from the same culture as the assembly; reads from a different stock of the same strain introduce that stock's variants.
• Ion Torrent reads are not a finishing polisher for Nanopore assemblies: they fix most errors but leave the shared homopolymer indels (92–98% of their residue) — use Illumina to finish.

Running the benchmark

conda activate HashBrown
bash validation/benchmark_polishers.sh data/ results/ 16

data/ holds one folder per sample: reference.fasta (the gold standard), the Nanopore fastq and the Illumina pair — see validation/README.md for the exact layout and how to simulate reads (badread + wgsim) when you have no finished genome of your own.

The harness assembles each sample once, then resumes a copy of that state for each of the 8 polisher combinations, compares every final assembly with the gold standard using QUAST, and writes benchmark_results.tsv: mismatches and indels per 100 kbp, misassemblies, NGA50, length and contig count per combination.

Reading the results

• The best combination has the fewest mismatches + indels per 100 kbp and no misassemblies.
• Indels are the Nanopore signature error; if none → polypolish+pypolca barely beats medaka → polypolish+pypolca, Medaka is not paying for its runtime on your basecalls.
• Check the low-coverage samples separately: that is where polishers turn harmful, and a mean over high-coverage samples hides it.

If your measurements disagree with HashBrown's defaults, open an issue with the table — the defaults follow the evidence.