Skip to content

eidolon v3.3.0

Choose a tag to compare

View all tags
@joshfactorial joshfactorial released this 01 Sep 04:34
· 320 commits to main since this release
v3.3.0
32fc390
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

eidolon v3.3.0 — fragment length realism

gen-frag-length-model now keeps the fragment length distribution it measures, instead of collapsing it to a mean and a standard deviation. The shipped default was rebuilt from real sequencing data, with its provenance recorded for the first time.

If you build your own fragment models, or rely on the default, this changes your output.

Built models keep their shape

The builder computed a full histogram of observed template lengths and then discarded it to fit a normal distribution.

A normal is symmetric by construction. Every real size-selected library is right-skewed — a steep left edge from size selection, and a long tail of larger fragments that escaped it. That tail is exactly what a paired-end SV caller thresholds against when deciding an insert is "larger than expected", and it was being thrown away at the point of fitting.

Measured against the BAM it was built from — HCC1395 normal, 16.3M read pairs:

mean sd skew p99
new (discrete) 0.01% 0.03% 0.001 0.00%
old (normal fit) 0.29% 2.68% 0.326 4.81%

The old fit matched the mean and standard deviation and missed the shape entirely, under-predicting long fragments by 4.81% at p99.

Cross-validated across chromosomes. A model built from chr20/21/22 checked against chr1 of the same library — 27.6M independent read pairs — agrees to within 0.34% on the mean and 0.011 on skew. Fragment length is a property of the library preparation, not of the region.

Verified end to end. Simulating with a trained fragment model, insert standard deviation, skew and p99 all measure 1.0x of real, closed from 1.3x, 10.7x and 1.2x.

The shipped default was replaced

The previous default was left-skewed (−0.434) where every real library is right-skewed. It was centred 130 bp high, truncated at 799 bp, had 33 integer lengths inside its own range with no bin at all, and carried an isolated spike at fragment length 1. Its provenance was unknown; it predates the Rust port, and nothing tested it — the test that existed pinned all 766 of its values as a literal array, which asserted the bytes had not changed and never that they were usable.

The new default comes from HCC1395 matched normal (SEQC2 WGS_NS_N_1, NovaSeq, GRCh38 chr20/21/22, 32.6M read pairs):

1087 bins over 8–1094 bp, no gaps
mean 431.8   sd 112.3   skew +0.528
p05/p50/p95/p99 = 258/424/623/746

Full provenance is in eidolon-core/src/models/model_data/README.md, which also records that the other five shipped models still have none.

Treat it as one real library, not a universal shape

Fragment length is set by chemistry — size selection, bead ratios, kit version. Insert skew measured 0.521 on this library and 0.122 on NA12878: 4x apart between two real human libraries.

If your data differs, build your own. It is a two-minute job and the whole reason the command exists:

eidolon gen-frag-length-model -c your_config.yml
# or, with GC bias, from one BAM walk:
eidolon gen-bam-models -c your_config.yml

Compatibility

  • Model files change format, from two floats to a distribution. Existing model files still load; models you rebuild will differ from ones built with v3.2.1.
  • distribution: normal restores the previous two-parameter fit. It remains the right choice for sparse input — exome, amplicon, a small targeted BAM — where there is not enough data to estimate a shape. Below 10 distinct observed lengths the discrete builder refuses rather than smoothing a handful of spikes into a plausible-looking curve.
  • min_reads no longer deletes individual lengths. Deleting a sparse bin is what punched holes in the distribution, and it bit hardest in the tail. It is now a floor on the total number of observations; sparse bins are handled by smoothing instead.

Also in this release

Sparse histograms are smoothed so the support has no holes, with the bandwidth from Silverman's rule — sparse input gets more smoothing, and a dense whole-genome histogram is returned untouched. "No gaps" is an asserted postcondition, not a hope.

The builder also no longer materialises one array element per read pair. Everything is computed from the histogram, so a whole-genome BAM costs about 12 KB instead of ~18 GB and two sorts of 800M elements.

Everything else since v3.2.1 is validation harness (scripts/delta/) and documentation. None of it ships in the binary or the conda package.

Assets

Five platform binaries: x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-rhel8, aarch64-unknown-linux-gnu-rhel8, x86_64-apple-darwin, x86_64-pc-windows-msvc.exe.

All merged pull requests
  • feat(delta): realism panel — measure the gap between eidolon reads and real human data — #636
  • chore: a release skill, so the post-release steps stop being re-enumerated — #640
  • chore: conda sha256 for v3.2.1 — #641
  • chore: merge main into develop after v3.2.1 — #642
  • chore: merge main into develop — conda sha256 for v3.2.1 — #643
  • fix(delta): the realism panel could not run on any acrocentric chromosome — #644
  • feat(models)!: the fragment builder keeps the measured shape instead of a Normal fit — #645
  • feat(delta): the realism panel can use trained models, and says which it used — #646
  • chore: regenerate the default fragment model from real data, with provenance — #647
  • docs(audit): the realism panel's first three runs measured the panel — #648
  • feat(delta): the realism panel can turn adapter read-through on — #649
  • feat(delta): dump the candidate breakpoints, not just how many there were — #650
  • fix(delta): a candidate site's support could exceed its own depth — #651
  • feat(delta): measure where real indels sit, and whether they are variants or errors — #652
  • feat(delta): join candidate clip boundaries to nearby indels; load samtools — #653
  • feat(delta): align simulated reads to the reference the real BAM actually used — #654
  • fix(delta): the summariser matched its inputs by substring, and lost the background — #655
  • fix(delta): the realism panel loads its own tools instead of inheriting them — #656
  • chore(release): v3.3.0 — fragment length realism — #659

The two in bold are the only ones that change the shipped binary.

Full Changelog: v3.2.1...v3.3.0

Assets 7
Loading