BioForge v10.0.0 - Desktop app

v10.0.0 - Desktop app

BioForge now has a desktop app: the same engine behind a friendly local window, for people who don't write code. Everything runs locally and offline - your DNA never leaves your machine (DNA Edge).

Two faces, same code, same version:

• No Python (just double-click): download the BioForge-v10.0.0-windows.zip asset below, unzip, run BioForge.exe. Self-contained.
• From the package: pip install "bioforge[app]" then bioforge-app.

Five tabs, each with a plain-language explanation: Sequences, Quality (FastQC-style), Align, Nanopore (raw signal -> bases), Evolution.

The app ships inside the package (bioforge.app) and as the .exe. 546 tests green.

The Windows .exe is built automatically by CI and attached to this release.

v9.1.0 — nanopore basecaller: 70% → 74% on real R9.4

A measured accuracy improvement to the Level 7 nanopore basecaller (pure NumPy, no AI),
on real captured R9.4 signal (E. coli, n=36, vs the production Guppy basecall), plus a
robustness bug sweep.

What changed

• Iterative per-read rescaling: a first Viterbi pass tells us which k-mer each
  event is; the per-read scale is then refit (robust least squares) to the actual
  levels of those k-mers, correcting the read's composition bias.
• Tuned transitions (p_stay 0.42 → 0.50): the old value over-penalised the STAY
  states that over-segmentation produces.
• Measured, not assumed: per-k-mer emission noise, drift correction and EM model
  re-estimation were all tried and dropped (they didn't help) — the fixed classical
  model is at its ceiling. Clean ablation.
• Robustness hardening: bug sweep of the new module — fixed two edge-case bugs
  (empty-signal path handling; inf/nan inputs no longer crash the rescaler) and added
  adversarial regression tests.

Result

Mean ~74.5%, median ~74%, range 68–82% (was 70.3% in v9.0). Reproduce it:
python tools/bench_basecaller.py. Still the classical R9-era route, still far below
neural Dorado's ~99% — the honest ceiling of a fixed model; R10 remains out of scope.

Tests

525 passing.

v9.0.0 — Level 7: a from-scratch nanopore basecaller

BioForge now reads raw electrical signal from Oxford Nanopore devices and turns it
into bases with a classical basecaller built entirely from scratch — pure NumPy, no
neural network, no GPU. The same algorithmic family (Viterbi dynamic programming) as
the Level 3 aligner.

What's new

• Signal I/O: read_pod5 (modern) and read_fast5 (legacy) readers. The only
  place an optional dependency is used (pip install "bioforge[nanopore]"), purely
  to open the file format.
• Pure-NumPy science: normalisation, event detection, our own pore-model
  estimation (estimate_pore_model — we learn the table, not copy it), and a Viterbi
  decoder with stay/step/skip states (viterbi_basecall) plus per-read
  moment-scaling.
• basecall(): raw signal to bases in one call.

Honest, reproducible numbers

On real captured R9.4 signal (E. coli, n=36, identity to the production Guppy
basecall): mean ~70%, median ~71%, range 63–77%. That is in the range of the
historical classical basecallers (nanocall ~68–85%) and far below the ~99% of the
neural Dorado. That gap is the honest point: the classical route is an R9-era method
and, without AI, it has a ceiling. Reproduce it: python tools/bench_basecaller.py.

Modern R10 chemistry is out of scope by design (9-mers → 262,144 hidden states make
an O(T·states) Viterbi infeasible on a laptop, and ONT ships no flat R10 k-mer table)
— which is the real reason the field moved to neural basecalling.

Requirements

• Python >= 3.10, NumPy. For nanopore signal I/O: pip install "bioforge[nanopore]".

Tests

523 passing.

v8.0.0 — Level 6: honest predictor judge + reality filter

The biggest update in BioForge's history. Two judgment tools (not predictors)
and a safety net against silent data corruption.

What's new

Level 6 — the honest judge (EvolutionBenchmark)

Judge any evolution predictor honestly. Every check exists because it caught us:

• Trivial-baseline bar — the rival is not 0.5, it's the best free axis (frequency,
  site mutability, conservation). Our own "AUC 0.80" turned out to be the mutability axis.
• Novel-mutation regime — separates already-circulating mutations (where counting is
  enough) from genuinely new ones.
• Bootstrap 95% CI — small wins that evaporate on resampling are exposed.
• Pretraining-leakage detector — flags a model that remembers instead of predicting.

Judged on real H3N2 flu: our trained model scores AUC 0.837 global / 0.631 on novel
mutations, over the trivial mutability bar (0.793).

Level 6 — the reality filter (RealityCheck)

Of the "concerning" mutations another tool (EVEscape, ESM-2, a DMS assay) hands you,
which ones have real traction in the population? Plugs in behind any of them.
Two tiers, never mixed: OBSERVED (evidence, AUC 0.97) vs ESTIMATED (conjecture, AUC 0.72).

Data-integrity safety net

• An anti-corruption guard that refuses to mis-encode a mistyped sequence
  (raises instead of silently turning it into N).
• Property-based invariants over both DNA and protein alphabets.
• tools/integrity_check.py — a reusable integrity certificate.

Critical bug fixed

The multiple-sequence aligner was packing every protein as DNA, silently turning
non-ACGT residues into N. It corrupted the whole Level 5 foundation. Fixed, model
retrained on clean data, every number re-measured. The two new Level 6 tools are
what caught it.

Requirements and install

• Python >= 3.10
• Only dependency: NumPy (the C engine and the trained model ship pre-compiled)

pip install bioforge
Optional deep-learning axis (ESM-2), not needed for the core:


pip install "bioforge[ai]"
Tests
498 tests passing — including property-based integrity tests over both alphabets
and the full Level 6 judge/filter battery.

Command-line tools
bioforge-analyze · bioforge-qc · bioforge-bgzip · bioforge-evolution

BioForge v7.0.0 — The Evolution Front

The largest release in the project's history. An entire new level (L5), a trained model, and a command-line tool — genome-agnostic, measured, and with honesty baked in.

What's new
Predict which mutations will rise — bioforge-evolution
Given dated sequences of a gene under selection (a flu HA across seasons, a gene under pressure), BioForge now ranks the mutations most likely to rise next:

bioforge-evolution rank strains.fasta --top 20 # rank candidate mutations
bioforge-evolution rank strains.fasta --novel # only never-seen mutations
bioforge-evolution backtest strains.fasta # is it better than "tomorrow = today"?
bioforge-evolution lineages strains.fasta # designate stable lineages
Stable lineages without a phylogenetic tree
designate_lineages builds Pango/autolin-style stable lineages using a Genotype Representation Index computed from two matrix multiplies over the MSA — no IQ-TREE, no cluster. It runs on a laptop where phylogenetic methods need a server.

A trained model that runs in pure NumPy
The mutation ranker is a small neural net (MLP 2×64). PyTorch was only the scaffolding — the shipped model is a 39 KB .npz and three matrix multiplies. No PyTorch, no GPU. It beats a plain linear model on all six held-out tests, and helps most when generalizing to an influenza type it has never seen (cross-virus).

Optional ESM-2 axis — with its leakage measured, not hidden
pip install bioforge[ai] adds a protein-language-model viability axis. Its pretraining leakage is documented (AUC drops ~0.20 on data after its training cutoff), and it is off by default.

Requirements
Python 3.10 or newer
NumPy 1.24 or newer — the only required dependency
Windows, Linux or macOS — the C engine ships pre-compiled; if it is unavailable on your platform, BioForge falls back to a transparent NumPy implementation
Optional (bioforge[ai]): PyTorch 2.0+ and Transformers 4.30+, only for the ESM-2 viability axis. The core tool and the trained ranker need none of this.
Install

pip install --upgrade bioforge # core
pip install --upgrade "bioforge[ai]" # + optional ESM-2 viability axis
Input format
The evolution tools read a FASTA where each record carries a date in its header — a year, or YYYY-MM for month resolution:

A/Sydney/5/2021|2021-03
MKTIIALSYIFCLVFA...
strain_2019
MKTIIALSYIFCLVFA...
rank works on protein; pass --translate to translate nucleotide input first. Records without a recognizable date are skipped.

The honest part (this is a feature)
None of this is scientifically novel — DERIVE, EVEscape and Hie et al. already exist and are better, with more resources. BioForge's value is not beating the state of the art. It is being the integrated, accessible, honest box that runs on humble hardware and tells you its own uncertainty instead of selling hype.

Predicting exact frequencies is a dead end — it ties the naive "tomorrow = today" baseline at every horizon tested. So the tool ranks mutations instead, which is what the field actually measures.
The physico-chemical "escape" axis turned out inverted — in flu HA the substitutions that rise are conservative. It measures viability, not escape. Replicated across H3N2, H1N1 and B.
Every limit in this release is measured with confidence intervals and written into the README.
By the numbers
454 tests passing
Zero dependencies at inference (NumPy only; the C engine and trained ranker ship pre-compiled)
Mutation ranking: cross-virus AUC approximately 0.77–0.95 on flu HA
Trained model: 39 KB, three matrix multiplies, laptop-runnable
Documentation and changelog
Full documentation: README
Full changelog: v6.3.0...v7.0.0

BioForge v6.3.0 — Multiple sequence alignment (MSA)

A new tool in the box, and the foundation of the evolution front (strain
prediction needs sequences aligned by position; so do phylogeny and selection).

── Install ──────────────────────────────────────────────────
pip install bioforge

Requirements

• Python >= 3.10
• NumPy >= 1.24 (the only runtime dependency — no Biopython, no heavy deps)
• The C engine ships PRE-COMPILED inside native wheels for Windows, Linux and
  macOS — no compiler needed. Falls back to the pure-NumPy path elsewhere.

Added

• bioforge.msa.align_multiple(sequences) -> MSAResult — multiple sequence
  alignment via the center-star heuristic: align every sequence to a central one
  (using the C aligner), then merge by propagating gaps. Ideal for sets of
  similar sequences (e.g. the same gene across strains over time).
  MSAResult.consensus() gives the majority consensus.
• Public API: from bioforge import align_multiple, MSAResult

Honest scope

• Center-star is the simple, correct starting point, best for SIMILAR sequences.
  Serious aligners (Clustal Omega, MAFFT, MUSCLE) use progressive alignment +
  iterative refinement, better for divergent sets — a planned future upgrade.

Usage
from bioforge import align_multiple
msa = align_multiple(["ATGGCCTTAGGCTA", "ATGGCGTTAGGCTA", "ATGGCCTTAGCTA"])
for row in msa.aligned: print(row)
print(msa.consensus())

Tests

• 14 tests, incl. the key property: removing gaps from any row reproduces the
  original sequence exactly (no data loss). 375 tests total.

BioForge v6.2.2 — Mapping accuracy validated (99.8% on real E. coli)

Fast is worthless if it's wrong — so here's the proof it isn't. No engine
changes; a new benchmark tool + the result in the README.

── Install ──────────────────────────────────────────────────
pip install bioforge (Python >= 3.10, NumPy only)

Added

• tools/accuracy_vs_minimap2.py — measures mapping ACCURACY on a real genome:
  simulates reads recording each read's true origin, maps with BioForge and
  minimap2, and counts how many land at the correct position (±tolerance). Real
  genomes have repeats — the hard case.

Result (E. coli K-12, 4.64 Mb, 5000 reads, ±50 bp)

• 5% error: BioForge 99.8% correct · minimap2 99.8% · 99.8% concordance
• 10% error: BioForge 99.7% · minimap2 99.9%
• As accurate as minimap2 — not fast-at-the-cost-of-correctness. Honest note: at
  higher error minimap2 is marginally ahead, and this is E. coli scale.

BioForge v6.2.0 — int16 SIMD extension (narrows the single-thread gap)

The anti-diagonal banded kernel gains a 16-bit variant that processes twice the
cells per instruction when scores fit (reads <= 12,000 bp); longer reads stay on
int32. Multi-core stays on par with minimap2.

── Install ──────────────────────────────────────────────────
pip install bioforge

Requirements

• Python >= 3.10
• NumPy >= 1.24 (the only runtime dependency — no Biopython, no heavy deps)
• The C engine ships PRE-COMPILED inside native wheels for Windows, Linux and
  macOS — no compiler needed. Falls back to the pure-NumPy path elsewhere.

── What changed ─────────────────────────────────────────────
Added

• _nw_banded_diag_simd_i16: AVX2 int16 (16-lane) version of the banded
  anti-diagonal kernel, with a 16x int16 reversal (per-lane shuffle + half swap).
  A dispatcher routes by size: int16 (m,n <= 12000) -> int32 (8-lane) -> scalar
  (no AVX2). Bit-identical to the scalar kernel (same DP and tie-break).

Performance (honest — WSL, 4.8 Mb, 6000 reads, 5% error, minimap2 -a)

• Extension kernel 2000x2000: 1.42x over int32 (the theoretical 2x is diluted by
  fixed costs — malloc, traceback, edges — that don't vectorise).
• 1 thread: BioForge ~1.87 vs minimap2 ~2.2 Mb/s -> ~1.18x behind (was ~1.3x).
• 4 cores: on par / ahead (~4.3-5.0 vs ~4.3-4.9). Both map all 6000.
• The rest of the single-thread gap is split between seeding (~25%) and chaining
  (~34%); closing it fully is diminishing returns.

Tests

• int16<->core kernel parity clean (incl. N, edges, narrow band); valgrind clean
  (0 errors / 0 leaks) on the int16 path with its 16-lane reversal. 361 tests.

BioForge v6.1.0 — Columnar map_batch (multi-core on par with minimap2)

The Python serial tail that rebuilt Mapping objects is gone, so multi-core
scaling is no longer capped. On 4 cores, BioForge is now on par with minimap2
on the reference benchmark (within run-to-run noise, sometimes ahead).

── Install ──────────────────────────────────────────────────
pip install bioforge

Requirements

• Python >= 3.10
• NumPy >= 1.24 (the only runtime dependency — no Biopython, no heavy deps)
• The C engine ships PRE-COMPILED inside native wheels for Windows, Linux and
  macOS — no compiler needed. On any other platform it falls back to the pure
  NumPy path automatically (same results, slower).
• Building from source on an unsupported platform needs GCC:
  python bioforge/engine/build.py

Quick check
python -c "import bioforge; print(bioforge.version)"

── What changed ─────────────────────────────────────────────
Changed

• bio_map_batch now writes into a NumPy structured array (same layout as the C
  MapOut struct, verified offset by offset) instead of a ctypes array. The
  Python cover builds Mapping objects by reading each field as a column
  (.tolist(), C-level) — no per-field ctypes access, no intermediate dicts.
  Results are identical (guaranteed by test_cmap_parity).

Benchmark (WSL, 4.8 Mb genome, 6000 reads, 5% error, minimap2 -a)

• 4 cores: minimap2 ~4.0-5.7 vs BioForge ~4.2-4.7 Mb/s -> on par (the pure C
  engine already did 4.85; the serial tail dropped it to ~3.8, now ~4.4-4.7).
• 1 thread: still ~1.2-1.3x behind (minimap2 ~2.2 vs BioForge ~1.8 Mb/s).
• Both map all 6000. At E. coli scale; minimap2 may pull ahead at larger scale.

Tests

• 361 tests; map_batch identical across 1/2/3/4/all threads and equal to
  sequential map().

BioForge v6.0.0 — SIMD extension + multi-core scaling

The genome mapper becomes competitive. Measured head-to-head against minimap2
on the same machine (WSL): from ~4x behind to ~1.3x, both single-threaded and
on 4 cores, mapping the same reads. Not promoted yet — this is an honest
measurement, not a headline.

Added

• AVX2-vectorised banded extension. The alignment DP is traversed by
  anti-diagonals (independent cells) and processes 8 int32 cells per
  instruction. The kernel goes from 88 to 529 M cells/s (6x); since the
  extension is ~88% of mapping time, the full mapper is ~4x faster
  single-threaded. Bit-identical to the scalar kernel (diag>up>left tie-break
  replicated exactly); automatic scalar fallback when AVX2 is unavailable.

Fixed

• Real multi-core scaling in bio_map_batch: the OpenMP thread count was not
  being reset, so after a single-threaded call, later calls stayed on one
  thread and map_batch appeared not to scale. Now always set (n<=0 -> all
  cores). Result: ~2.3x on 4 cores.

Benchmark (honest — WSL, 4.8 Mb genome, 6000 reads, 5% error, minimap2 -a)

• 1 thread: minimap2 ~2.4 vs BioForge ~1.8 Mb/s (~1.3x)
• 4 cores: minimap2 ~4.0 vs BioForge ~3.0 Mb/s (~1.3x)
• Both map all 6000. At E. coli scale; minimap2 may pull further ahead at
  human-genome scale (not measured). A Python result-reconstruction tail
  remains (the pure C engine already does ~4.85 Mb/s).

Tests

• SIMD<->scalar parity 0/10,000 (incl. pathological narrow band); valgrind
  clean (0 errors / 0 leaks) on the SIMD kernel; map_batch identical across
  1/2/3/4/all threads and equal to sequential map(). New tools/bench_vs_minimap2.py.
  359 tests.