v4.0.1 — config mistakes report errors, and runs start faster

Two fixes. Simulation output is unchanged at the same seed: FASTQs, truth VCF and BAM
records are byte-identical to v4.0.0 on H1N1 and E. coli.

Fixed

• A bad config is reported as an error, not a crash (#842). A missing or malformed
  setting, a missing file, an output that already exists, or an unreadable path now stops the
  run with exit status 1 and a message naming the key or file. These used to panic: exit
  status 101 and thread 'main' panicked at …, sometimes with nothing more than "Error
  generating configuration!". Examples of the new messages: reference is required and has no default, the name of an existing output with the overwrite_output: true hint, and a missing
  config file reported against the subcommand that was asked for (it used to read as a
  filter-reads error for every subcommand). A model file that cannot be opened is also an
  error naming the file (#846).
• Every run starts about 0.6 s sooner (#845, #846). The default sequencing error model is
  20.8 MB of JSON once decompressed, and it was parsed through a streaming reader that cost
  ~720 ms per run. Models are now decompressed into memory and then parsed: startup went from
  0.72 s to 0.09 s on a small reference. The saving is per run, so it matters most for short
  runs and many small jobs. Measured on Delta, single thread, 10x: E. coli 12.28 → 10.99 s,
  yeast 24.03 → 22.94 s, chr22 68.20 → 67.44 s.

Docs

• The install instructions no longer promise a Bioconda package (#849). conda install -c bioconda eidolon finds nothing: the recipe is in review at
  bioconda-recipes#67420. Use a
  release binary until it is merged.
• CITATION.cff cites both NEAT papers (#850).

Evidence

• Output identity: v4.0.0-era develop and the #846 build write byte-identical FASTQ R1/R2, VCF
  body and BAM records on H1N1 and on E. coli at 5x.
• tests/config_errors_exit_cleanly.rs requires exit 1, the reason on stderr and no
  "panicked" for each subcommand's config errors.
• New model-loading tests compare the new loader against the old streaming parse on the shipped
  default, and refuse non-gzip input, a truncated stream and a stream that fails its checksum.
  Each test was checked by mutation.

Not verified

• Subcommands outside the three config loaders were not audited case by case for user-input
  panics beyond the shared output-file creation, which is fixed.
• The Delta timings above come from one run during a possible Delta filesystem incident. Only
  chr22's replicates agree closely (1% spread); E. coli's first replicate read 15.3 s.
• The ~40 MB of fixed memory noted in #670 is unchanged and still unexplained.

---

Release record and verification: #852. Bioconda: not yet available; the recipe is in review at bioconda-recipes#67420. Use the binaries below.

eidolon v4.0.0

eidolon v4.0.0 — stricter inputs, and fixes that change output

A major version because three things you may depend on now behave differently: config files with keys a subcommand does not read are refused, the rneat command is gone, and gen-mut-model refuses a VCF that disagrees with its reference. Each stops a run that used to complete, and each says what to change. See Upgrading from 3.x.

The rest are fixes, several of which change output for the inputs they affect: soft-masked references, input_vcf deletions next to symbolic SVs, capped sequencing-error fits, and the bundled tumor models.

Output is otherwise unchanged at the same seed. Checked on H1N1 (uppercase reference, no input_vcf): v3.5.1 and v4.0.0 write byte-identical FASTQs and truth VCF, and the same BAM records apart from the new NM tag.

📖 Documentation: https://ncsa.github.io/eidolon/

Compatibility (breaking)

• Unknown config keys stop the run (#496). Every top-level key, and the nested adapters:, frag_length: and gc_bias: sections, must be one the subcommand reads. The error names the key, the nearest accepted key, and the full list; for a NEAT config, gen-reads names the eidolon equivalent where one exists. An unread key used to be ignored silently: tumor_mutation_model: (meant as tumor_model:) simulated the tumor with the germline model and reported success. Every rejected key had no effect, so the fix is to remove or rename it. The release shakeout itself had been writing fragment_length_model:, which nothing read; it now writes fragment_model:.
• The rneat command is removed (#710). It was renamed to eidolon in v2.0.0 and kept as a deprecated alias. Release binaries never included it; conda installs did. Invoke eidolon.
• gen-mut-model refuses a VCF whose REF disagrees with the reference (#826). It stops at the first SNP or indel whose REF does not match the reference there, names the record and both bases, and gives the command to drop such records deliberately: bcftools norm -f reference.fa --check-ref x in.vcf.gz -Oz -o checked.vcf.gz. NEAT2 stopped the same way for SNPs. eidolon used to warn and skip a mismatched SNP but still count it toward the mutation rate, and never checked indels. A SNP at a contig edge is now also left out of the rate, as in NEAT2. A bed_file that names no reference contig now says so, listing both sets of names, instead of failing with "Unknown error".

Fixed: changes to generated output

• Soft-masked references simulate exactly like uppercase ones (#790). Lowercase bases now parse as plain bases. Context-weighted SNP placement used to miss the model in masked sequence, roughly half of a UCSC-style human reference, and fall back to the mean weight; SV REF and breakend anchor bases were written lowercase in the truth VCF.
• A literal deletion crossing a symbolic SV's boundary no longer inflates coverage (#691). When a deletion of 50 bp or more ran past the end of its sub-region, reads were drawn from the deletion to the end of the contig: +48% R1 reads on the ecoli test case. Affects input_vcf runs that combine literal deletions with <DUP>/<DEL>/<CNV> records.
• The bundled COSMIC tumor models are rebuilt (#761) with the current builder. Only the per-context SNV placement weights moved, by −7.4% to +12.9% per context, so gen-cancer-reads places somatic SNVs slightly differently. All seven were run end to end: each model's simulated spectrum matches its own weights (r ≥ 0.997).

Fixed: model fitting

• max_reads samples the whole FASTQ (#721). gen-seq-error-model took the first N records, which come from one tile of one lane; it now keeps a uniform, reproducible sample across the file. A capped fit therefore reads the whole file and takes longer, and it gives a different model, one that describes the whole library.
• A one-sided indel VCF keeps its fitted lengths (#766). An insertions-only (or deletions-only) training VCF used to discard both fitted length distributions; now only the unobserved type uses its default, with a warning.

Added

• Golden BAM records carry NM (#536), the edit distance to the reference, counted as samtools calmd counts it. samtools stats and MultiQC used to report a 0 error rate for any simulated BAM.

Fixed: errors and crashes

• filter-reads accepts plain .fastq/.vcf and dotted directories (#824). A plain file panicked with a subtraction overflow, and a dot in a directory name rebuilt the wrong path. Output is always gzip-compressed, so a plain input's output gains .gz.
• gen-seq-error-model names the cause when a BAM's MD tags disagree with its reads (#767), with a samtools calmd fix, instead of failing as a bare InvalidWeights. A mate population with no bases is refused by name.
• A correct breakend no longer warns about a missing END/SVLEN (#497). A <DEL> or other span-requiring SV without one still does.

Testing

A test-adequacy audit (#819) made 22 unit tests assert content instead of existence, tested check_query_denominator, and added mutation checks across the decisions behind quoted numbers; two gaps it found are pinned (compare-vcfs requires the ALT to match; mutation-model variant shares follow observed counts). Four sv_pipeline.sbatch functions remain untested (#825).

Not verified for this release

• The gen-mut-model REF check (#826) and the filter-reads fix (#824) were tested on fixtures, not run on real data.
• De novo breakends share #497's code path but no test runs them.
• The rebuilt tumor models' SV components were not exercised end to end (#761 reports they did not change).
• Delta-only scripts were checked for unknown config keys by search, not by running them (#496). A Delta script carrying an unread key now fails at job start, naming the key.
• The straddling-deletion fix (#691) was not rerun on Delta. Known limitation: deleted bases past the sub-region boundary still get reference coverage.
• Uniform max_reads sampling (#721) was not compared against head sampling on HG002, and a capped run was not timed, so the HPC chapter's 1:30:00 estimate is not measured.

---

Release testing is recorded on #816. Full detail in CHANGELOG.md.

Full Changelog: v3.5.1...v4.0.0

eidolon v3.5.1

eidolon v3.5.1 — release binaries run under their download name

One change from v3.5.0, in how the binary picks its command. Simulation output, models and every other behavior are unchanged.

📖 Documentation: https://ncsa.github.io/eidolon/

Fixed

• A release binary now runs under any file name (#794). Every release since v2.0.0 stopped with unrecognized subcommand 'eidolon-x86_64-unknown-linux-gnu' when run as downloaded, because the binary reads the name it was invoked by to choose its command. Any name that is not eidolon or a subcommand now runs as eidolon.
• A binary named after a subcommand, such as gen-reads, still runs that subcommand directly. The rneat alias is unaffected.
• The install page now says the rename is only needed for releases before v3.5.1.

Evidence

A new test copies the real built binary to each release asset's name and to an arbitrary name, and requires the same --version as eidolon. Before the fix it failed with the release error; it passes now. A second test requires a binary named gen-reads to still run gen-reads, and a mutation check confirmed it catches an over-eager fallback.

Not verified: the Windows .exe asset. The name is compared on its file stem, so it should resolve, but it has not been run on Windows.

---

The v3.5.0 release notes cover the germline defaults this patch ships with. Full detail in CHANGELOG.md.

Full Changelog: v3.5.0...v3.5.1

eidolon v3.5.0

eidolon v3.5.0 — germline defaults fitted from one human genome

The shipped germline defaults now come from GIAB HG002, the library the quality model already came from, instead of a mix of NEAT2 values and a second sample. The sequencing-error substitution matrix is fitted from current NovaSeq data, using a new method that cancels the sample's own variants.

Output differs from v3.4.0 at the same seed. Re-baseline anything measured against 3.4.0.

📖 Documentation: https://ncsa.github.io/eidolon/

New defaults

default	now	was
mutation rate, SNP/indel shares, homozygous fraction	HG002 v4.2.1 truth VCF in its high-confidence BED: 0.0015162, SNP/ins/del 0.873/0.062/0.066, 0.388 hom	NEAT2 NA12878: 0.0010987, 0.95/0.03/0.02, 1/3
trinucleotide contexts, variant indel lengths	the same HG002 fit	NEAT2 NA12878
fragment length	HG002 2x250 BAM: mean 408.5, sd 93.2	HCC1395 NovaSeq: 431.8, 112.3
substitution matrix	mean of two NovaSeq 6000 libraries, fitted from mate-overlap disagreements	NEAT2 SSE_PROB, no recorded source

The mutation fit reproduces bedtools counts of the truth VCF exactly: 3,364,039 SNPs, 237,674 insertions and 252,894 deletions. GC bias stays uniform by default; the HG002 GC fit is held for named presets (#576).

Fixes that change generated reads

• A fragment's two mates now carry the same allele (#780). Before, 73 of 170 het pairs disagreed; now 0 do.
• SV junction R2 reads use R2's quality model (#753) and apply substitution errors in reference orientation (#777), like every other R2.
• Soft-masked references are counted as their uppercase bases in GC weighting and SV insertion composition (#771). Uppercase references are unaffected.
• A distribution with no mass is refused (#760) instead of sending every draw to its first value.

Fixes in the tools

• VCF/BED off-by-one (#770): compare-vcfs scoring and attribution, filter-reads and gen-mut-model compared 1-based POS to 0-based BED intervals.
• gen-mut-model labels G and T correctly (#758), drops its unused 4×4 matrix (#763), and refuses a VCF whose SNPs are all unusable (#760).
• gen-gc-bias-model fits medians, interpolates sparse bins, writes a per-bin report and refuses an uncovered BAM (#774).
• gen-cancer-reads defaults read_len to 250 like gen-reads (#754). gen-reads warns when it rescales a quality model (#742).

Fit your own substitution matrix

gen-seq-error-model with bam_file: now fits from mate-overlap disagreements by default (bam_method: overlap). The sample's variants, PCR errors and damage appear in both mates and cancel; only sequencing errors remain. known_variants_vcf: masks a VCF's sites. For a library whose mates don't overlap, use bam_method: mismatch.

Compatibility

• A mutation model written by 3.5.0 does not load in an older eidolon. It fails with a "missing field" error, not silently. Older models load here.
• gen-mut-model's transition_matrix_file is removed. A set value stops the run with an explanation; gen-seq-error-model's option of that name is unchanged.
• gen-seq-error-model's bam_method now defaults to overlap. Use mismatch to reproduce a 3.4.0 fit.

Evidence and limits

Validated on real data on NCSA Delta. The HG002 realism panel on the new defaults shows no regression against the previous run. The mutation refit matches bedtools. The NovaSeq fits rest on 3.0M and 656k counted errors. Not re-run: cancer simulation; the COSMIC tumor-model rebuild is #761. Open realism gaps are tracked in #692, #658 and #696.

---

Full detail in CHANGELOG.md. Model provenance in model_data/README.md.

Full Changelog: v3.4.0...v3.5.0

eidolon v3.4.0

eidolon v3.4.0 — a quality model measured from real, named data

eidolon's shipped sequencing-error model no longer comes from a sample nobody recorded. It is fitted from GIAB HG002 NIST_Illumina_2x250bps — public, downloadable, and documented down to the chunk, the sampling stride, the read count and the job that produced it.

📖 Documentation is now a site: https://ncsa.github.io/eidolon/

The default model is fitted from data you can download

Read length	250 bp
Quality scores	31 observed levels, Q2–Q40, continuous
Reads fitted	3,391,610 per mate, sampled across the file
error_rate	0.003774, against the inherited 0.006638

It carries both mates separately, because they are measurably different — 2.07x on error rate, 2.36x on degraded fraction.

Reads now degrade the way real reads do

About one read in ten in real Illumina data loses signal toward the 3' end and stays collapsed. eidolon's quality model had no per-read state to represent that and produced essentially none of it.

	real	v3.4.0	previous single-population model
R1, last 50 bases below Q25	11.12%	11.95%	0.54%
R1, below Q20	4.92%	5.91%	0.00%
R2, below Q25	26.22%	26.36%	11.74%
R2, below Q20	12.48%	14.00%	0.32%

Fixed: R2's quality scores were emitted backwards

This affected every paired FASTQ eidolon has ever produced. R2's per-cycle quality ran in reverse — reads improved toward the 3' end where real reads get worse — and because errors are injected from each base's own quality, R2's sequencing errors were mirrored with it.

Also

• Sequencing-error indels were ~40x too frequent and uniform across the genome; corrected and now scaled by local homopolymer run length.
• Model files carry a provenance stamp, so a model from a newer eidolon is refused rather than silently misread.
• gen-seq-error-model takes its read length from the reads rather than the first record, and validates FASTQ structurally as it reads.

Upgrading

• gen-reads' read_len now defaults to 250 to match the model. gen-cancer-reads is unchanged at 151.
• If you relied on that default and set fragment_mean, check it still suits 250 bp reads. Paired-end fragments have always had to exceed the read length; that is unchanged, and only the default moved. Configs from the shipped template set read_len explicitly and are unaffected.
• Output for a given seed changes, R1 included. Fragment placement is untouched, but a sequencing error consumes extra random draws, so changing which R2 bases error shifts the stream. Per-seed content has never been part of the public API.
• Peak RSS rises ~22 MB on the default path: four quality tensors where there was one.

Known and open

somatic_snv_recall reads 0.8397 against a 0.921 baseline and has since #660, now on four builds. This release eliminated the quality model as the cause — it replaced the entire model and the metric did not move. Tracked in #680.

The default is HiSeq 2500-era chemistry with continuous quality scores, not binned — the best provenanced option available from GIAB's HG002 path, not the most modern one. See #730.

---

Full detail in CHANGELOG.md. Model provenance in model_data/README.md.

eidolon v3.3.0

eidolon v3.3.0 — fragment length realism

gen-frag-length-model now keeps the fragment length distribution it measures, instead of collapsing it to a mean and a standard deviation. The shipped default was rebuilt from real sequencing data, with its provenance recorded for the first time.

If you build your own fragment models, or rely on the default, this changes your output.

---

Built models keep their shape

The builder computed a full histogram of observed template lengths and then discarded it to fit a normal distribution.

A normal is symmetric by construction. Every real size-selected library is right-skewed — a steep left edge from size selection, and a long tail of larger fragments that escaped it. That tail is exactly what a paired-end SV caller thresholds against when deciding an insert is "larger than expected", and it was being thrown away at the point of fitting.

Measured against the BAM it was built from — HCC1395 normal, 16.3M read pairs:

	mean	sd	skew	p99
new (discrete)	0.01%	0.03%	0.001	0.00%
old (normal fit)	0.29%	2.68%	0.326	4.81%

The old fit matched the mean and standard deviation and missed the shape entirely, under-predicting long fragments by 4.81% at p99.

Cross-validated across chromosomes. A model built from chr20/21/22 checked against chr1 of the same library — 27.6M independent read pairs — agrees to within 0.34% on the mean and 0.011 on skew. Fragment length is a property of the library preparation, not of the region.

Verified end to end. Simulating with a trained fragment model, insert standard deviation, skew and p99 all measure 1.0x of real, closed from 1.3x, 10.7x and 1.2x.

The shipped default was replaced

The previous default was left-skewed (−0.434) where every real library is right-skewed. It was centred 130 bp high, truncated at 799 bp, had 33 integer lengths inside its own range with no bin at all, and carried an isolated spike at fragment length 1. Its provenance was unknown; it predates the Rust port, and nothing tested it — the test that existed pinned all 766 of its values as a literal array, which asserted the bytes had not changed and never that they were usable.

The new default comes from HCC1395 matched normal (SEQC2 WGS_NS_N_1, NovaSeq, GRCh38 chr20/21/22, 32.6M read pairs):

1087 bins over 8–1094 bp, no gaps
mean 431.8   sd 112.3   skew +0.528
p05/p50/p95/p99 = 258/424/623/746

Full provenance is in eidolon-core/src/models/model_data/README.md, which also records that the other five shipped models still have none.

Treat it as one real library, not a universal shape

Fragment length is set by chemistry — size selection, bead ratios, kit version. Insert skew measured 0.521 on this library and 0.122 on NA12878: 4x apart between two real human libraries.

If your data differs, build your own. It is a two-minute job and the whole reason the command exists:

eidolon gen-frag-length-model -c your_config.yml
# or, with GC bias, from one BAM walk:
eidolon gen-bam-models -c your_config.yml

Compatibility

• Model files change format, from two floats to a distribution. Existing model files still load; models you rebuild will differ from ones built with v3.2.1.
• distribution: normal restores the previous two-parameter fit. It remains the right choice for sparse input — exome, amplicon, a small targeted BAM — where there is not enough data to estimate a shape. Below 10 distinct observed lengths the discrete builder refuses rather than smoothing a handful of spikes into a plausible-looking curve.
• min_reads no longer deletes individual lengths. Deleting a sparse bin is what punched holes in the distribution, and it bit hardest in the tail. It is now a floor on the total number of observations; sparse bins are handled by smoothing instead.

Also in this release

Sparse histograms are smoothed so the support has no holes, with the bandwidth from Silverman's rule — sparse input gets more smoothing, and a dense whole-genome histogram is returned untouched. "No gaps" is an asserted postcondition, not a hope.

The builder also no longer materialises one array element per read pair. Everything is computed from the histogram, so a whole-genome BAM costs about 12 KB instead of ~18 GB and two sorts of 800M elements.

Everything else since v3.2.1 is validation harness (scripts/delta/) and documentation. None of it ships in the binary or the conda package.

Assets

Five platform binaries: x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-rhel8, aarch64-unknown-linux-gnu-rhel8, x86_64-apple-darwin, x86_64-pc-windows-msvc.exe.

---

All merged pull requests

• feat(delta): realism panel — measure the gap between eidolon reads and real human data — #636
• chore: a release skill, so the post-release steps stop being re-enumerated — #640
• chore: conda sha256 for v3.2.1 — #641
• chore: merge main into develop after v3.2.1 — #642
• chore: merge main into develop — conda sha256 for v3.2.1 — #643
• fix(delta): the realism panel could not run on any acrocentric chromosome — #644
• feat(models)!: the fragment builder keeps the measured shape instead of a Normal fit — #645
• feat(delta): the realism panel can use trained models, and says which it used — #646
• chore: regenerate the default fragment model from real data, with provenance — #647
• docs(audit): the realism panel's first three runs measured the panel — #648
• feat(delta): the realism panel can turn adapter read-through on — #649
• feat(delta): dump the candidate breakpoints, not just how many there were — #650
• fix(delta): a candidate site's support could exceed its own depth — #651
• feat(delta): measure where real indels sit, and whether they are variants or errors — #652
• feat(delta): join candidate clip boundaries to nearby indels; load samtools — #653
• feat(delta): align simulated reads to the reference the real BAM actually used — #654
• fix(delta): the summariser matched its inputs by substring, and lost the background — #655
• fix(delta): the realism panel loads its own tools instead of inheriting them — #656
• chore(release): v3.3.0 — fragment length realism — #659

The two in bold are the only ones that change the shipped binary.

Full Changelog: v3.2.1...v3.3.0

v3.2.1

v3.2.1 — complete release assets

The binary is unchanged from v3.2.0: eidolon/src, eidolon-core/src and their
manifests are byte-for-byte identical. This tag exists to publish the macOS
binary v3.2.0 never got, with all targets built by the pinned 1.98.0 toolchain
rather than floating stable.

eidolon v3.2.0

v3.1.0 shipped with insertion limitations listed as known and documented. This release closes them, together with the wider defect they belonged to: records that reached the truth VCF and never reached the reads. Seven issues turned out to be one shape.

Fixed

#516	insertions longer than a read are sequenced end to end, not just their first ~150 bp
#589	the golden BAM CIGAR describes them, instead of recording junction reads as pure M
#590	literal deletions live on the haplotype, so they remove coverage
#498	BND ALTs carrying novel junction sequence splice it into the reads
#500	symbolic <INS> is realized; unpaired breakends are refused instead of destroying 41% of local coverage
#625	one 5 kb deletion no longer silences 22% of a contig
#603	supplied SVs come out of the de novo budget instead of adding to it
#591	REF == ALT and no-call genotypes are refused or normalized
#599	chimeric read generation is deterministic again
#583	a failing validation run can explain itself again

The first three are one architectural change: build the altered haplotype, then read from it. Insertions and deletions now share a coordinate map in which inserted bases occupy real space and deleted bases occupy none, so a fragment can begin inside an insertion and can never be placed across sequence that does not exist on the molecule.

Validated

Delta jobs 21575385 and 21603825 — chr20+21+22, 30x, purity 0.6, SV_RATE_SCALE=30, both exit 0:

• 11/11 insertions present in the reads, 9 longer than the 151 bp read, largest 968 bp
• 76/76 deletions with junction sequence present, 76/76 removing coverage, flank ratios 0.33–0.77
• no coverage holes at genome scale; scorer selftests pass on 153 + 78 records

Known limitations

• Insertion sequence is random, not mobile-element (#580). It is unique in the genome by construction, so every supporting read maps at MAPQ ≥ 30, where real Alu and L1 map ambiguously. INS recall measured against eidolon insertions is optimistic.
• INS denominators are thin. Both campaigns planted 9 at-risk insertions against a floor of 20. Those 9 worked; that is not insertion realization at scale.
• Single breakends are refused, not supported (#623) — a legitimate VCF 4.2 call.
• The CIGAR clip gate sits near its detection floor for somatic deletions (#630).
• Copy-number depth accuracy is size-dependent (#584). Accurate at 10 kb; biased high on kb-scale events, because junction reads are emitted on top of coverage-multiplied reads.

Validation harness

Most of the above was found by the harness, and several harness defects were found by its own gates disagreeing with each other. Four read-level evidence gates now run per campaign before BAMs are pruned — insertion sequence, deletion junction sequence, deletion coverage against flanks, and CIGAR clip support against flanking controls. Clippy is a blocking CI gate and the Rust toolchain is pinned.

Full changelog: 3.1.0...v3.2.0

eidolon v3.1.0

What's Changed

• fix(compare-vcfs): inherit header declarations from the source VCF (#444) by @joshfactorial in #461
• fix(delta): measure observed VAF without a genotype call (#450) by @joshfactorial in #463
• fix(delta): fail the signature check below the SNV floor instead of warning by @joshfactorial in #464
• fix(delta): calibrate the SV scorer, compare BND junctions as spans, drop the Python (#457) by @joshfactorial in #462
• fix(delta): scorers must prove they measured what they were given by @joshfactorial in #468
• fix(delta): BNDspan cannot measure junction recall — disable it on probe failure by @joshfactorial in #469
• fix!: de novo BND truth described a rearrangement the reads did not carry by @joshfactorial in #470
• docs: test-adequacy standard; fold v3.0.1 into v3.1.0 by @joshfactorial in #471
• fix(test): restore a silently disabled test; close the BND coverage gaps it hid by @joshfactorial in #472
• test: assert chimeric junction SEQUENCE, not just QNAMEs by @joshfactorial in #473
• test: assert configured purity actually reaches the merged read mix by @joshfactorial in #475
• test: assert the trained quality profile reproduces its POSITIONAL shape by @joshfactorial in #476
• fix(delta): report the binary version, and refuse a stale one by @joshfactorial in #477
• test: assert de novo SV span consistency and exact IUPAC constituents by @joshfactorial in #478
• feat(delta): normalise Manta's inversion representation before scoring by @joshfactorial in #479
• fix!: POS now means the same thing for every symbolic SV type (VCF 4.2) by @joshfactorial in #480
• test: cover INV junction geometry; the "~8% unexplained" was my measurement by @joshfactorial in #481
• test: close the last three vacuous assertions from the audit by @joshfactorial in #482
• feat: eidolon validate — check artifacts against what their consumers accept by @joshfactorial in #483
• feat(delta): gate emitted artifacts through eidolon validate by @joshfactorial in #484
• fix(delta): score BND against the RAW calls, INV against the converted ones by @joshfactorial in #485
• feat: port scn_af_compare.py to eidolon compare-af; closes #466 for that helper by @joshfactorial in #486
• feat!: sample BND geometry instead of emitting head-to-head only (#458 item 2) by @joshfactorial in #487
• docs(report): correct the BND account, extend the defect table, flag what is pending by @joshfactorial in #488
• test(delta): automated tests for the BND geometry harness; stratify BND recall by geometry by @joshfactorial in #489
• fix(delta): score the canonical SVTYPE list, not a glob of the truth directory by @joshfactorial in #490
• fix(core): emit adapter read-through as a soft clip, not an aligned match by @joshfactorial in #491
• docs: engineering audit of three months of AI-assisted development by @joshfactorial in #493
• docs: retract the translocation claims — eidolon has never generated one by @joshfactorial in #492
• docs(P2): measure the PCAWG SV corpus — TRA is 100% inter-chromosomal by @joshfactorial in #494
• test: CI-enforce the SV support matrix by @joshfactorial in #501
• feat(delta): pool truvari results across replicate SV runs by @joshfactorial in #503
• perf(delta): request the whole node's cores; --mem already reserves it by @joshfactorial in #504
• fix(delta): subvaf validation --mem 64G -> 96G; 64 was below the observed peak by @joshfactorial in #506
• fix(delta): decoy displacement must exceed the event; add the missing BND decoy by @joshfactorial in #508
• fix(delta): calibration controls that did not run must fail, and empty SV strata must be reported by @joshfactorial in #509
• fix(sv)!: tag de novo INS with SVTYPE/SVLEN so it is separable from a small indel by @joshfactorial in #510
• fix(delta): stale-binary detection by commit, inline BAM prune, and a disk pre-flight by @joshfactorial in #514
• fix(delta): recover the missed disk pre-flight; pin the large-insertion read gap (#516) by @joshfactorial in #515
• feat(delta): verify planted insertions reach the reads before the BAMs are pruned by @joshfactorial in #517
• fix(sv)!: cap de novo insertions at read_len-1; correct the #516 mechanism; retract the DUP-spanning defect by @joshfactorial in #518
• refactor(delta)!: retire the BND/BNDinv split (#507); verify the query denominator (#511) by @joshfactorial in #520
• fix(delta): scale the disk gate by genome size and coverage instead of a flat 214 GB by @joshfactorial in #521
• docs(delta): link with gcc, not the Cray cc wrapper — and say why by @joshfactorial in #522
• docs(readme): correct the NEAT comparison table and refresh the CLI help block by @joshfactorial in #523
• test(seq-error): assert the transition matrix's content, not that a file appeared by @joshfactorial in #525
• docs: expand CONTRIBUTING and add an ONBOARDING tour for teammates by @joshfactorial in #528
• docs: retire docs/session_handoff.md, migrating the one finding it uniquely held by @joshfactorial in #531
• test(bam): validate MD-tag transition counting against samtools calmd by @joshfactorial in #532
• fix(gen-seq-error-model)!: refuse to substitute the default transition matrix by @joshfactorial in #533
• test(bam): pin the MD reader against a real aligner's output (1000 Genomes HG00096) by @joshfactorial in #535
• docs: roadmap for rebuilding SV confidence from the reads up by @joshfactorial in #538
• docs(sv): record the whole-genome tier — the support matrix was H1N1-only by @joshfactorial in #539
• test(sv): Gate 2 for DEL — assert a real aligner produces the evidence a caller needs by @joshfactorial in #542
• test(sv): Gate 2 for DUP, and a measured mechanism for #499 by @joshfactorial in #543
• test(sv): Gate 2 for INV — and the reads are cleared of the precision artifact by @joshfactorial in #544
• test(sv): Gate 2 for BND — and mated breakends are measured copy-number neutral by @joshfactorial in #545
• docs: a mutation experiment must verify the mutation was applied by @joshfactorial in #547
• docs: split Gate 2 into 2a (realigned) and 2b (golden BAM) by @joshfactorial in #549
• fix(bam)!: golden BAM records were not in reference orientation (Gate 2b) — closes #550, #551 by @joshfactorial in #552
• fix(bam): compute TLEN from realized mate spans by @joshfactorial in #556
• docs: refresh GitNexus index metadata by @joshfactorial in #557
• fix(delta): fail when INS probes are not evaluated by @joshfactorial in #558
• test(bam): verify planted SVs reach golden BAM by @joshfactorial in #559
• docs: record Delta SV characterization baselines by @joshfactorial in #560
• fix(delta): report all-calls SV recall by @joshfactorial in #561
• fix: apply subclone CCF to structural variants by @joshfactorial in #562
• docs: record release candidate SV validation by @joshfactorial in #563
• release: v3.1.0 correctness and validation by @joshfactorial in #564

Full Changelog: v3.0.0...3.1.0

v3.0.0

eidolon v3.0.0

Two headline changes: intra-tumor heterogeneity (a tumor is now a mixture of subclones, not a single purity scalar), and a breaking rename of every token eidolon emits — completing the rusty-neat → eidolon rename begun in v2.0.0.

From this release eidolon follows Semantic Versioning 2.0.0 and declares a public API, so future version numbers mean something specific.

⚠️ If you have scripts that parse eidolon's VCFs, FASTQs or BAMs, read "Upgrading" before installing. Nothing about the reads or variants changed — only the names of emitted tokens. But a parser matching the old names will match nothing, silently, with exit code 0.

---

Upgrading from 2.0.0

What changed

Surface	v2.0.0 and earlier	v3.0.0	Migration
VCF INFO tags	NEAT_ORIGIN, NEAT_PROVENANCE, NEAT_REASON, NEAT_CCF, NEAT_VAF	EIDOLON_*	converter provided
VCF sample column	NEAT_simulated_sample	EIDOLON_simulated_sample	converter provided
FASTQ read names	@RNEAT_generated_*, RNEAT_chimeric_*	@EIDOLON_generated_*, EIDOLON_chimeric_*	not converted — see below
BAM read names (QNAME)	RNEAT_generated_*	EIDOLON_generated_*	not converted — see below

VCFs — convert them

tools/migrate_legacy_tokens.sh old_truth.vcf.gz new_truth.vcf.gz
tools/migrate_legacy_tokens.sh --check old_truth.vcf.gz    # report only; exit 10 = legacy tokens present

Idempotent, so it's safe to call unconditionally in a pipeline — it's a no-op on a file that's already current or isn't an eidolon VCF. It rewrites the header declarations and the record fields together, so the result is a valid VCF.

You cannot work around this with a dual-name bcftools filter. bcftools rejects an undefined tag when it parses the expression, so even

-i 'INFO/EIDOLON_ORIGIN="somatic" || INFO/NEAT_ORIGIN="somatic"'

fails outright on a file carrying only one of the two. Convert the file instead.

Read names — make your parser prefix-agnostic

Read names are not rewritten, in FASTQ or BAM. Rewriting files that are routinely hundreds of GB to change ~19 bytes per record isn't a reasonable ask, so eidolon filter-reads accepts both prefixes natively and warns once when it sees an old one.

Only the prefix changed. Everything after _generated_ — <contig>_<start>_<end>_<uniq>/<mate> — is byte-for-byte identical between versions. So rather than rewriting files, strip the prefix and your script works against either version, and against any future rename:

samtools view x.bam | cut -f1 | sed -E 's/^[A-Z]+_generated_//'
zcat x_r1.fastq.gz  | awk 'NR%4==1' | sed -E 's/^@[A-Z]+_generated_//'

Same for chimeric reads: sed -E 's/^@?[A-Z]+_chimeric_//'.

Audit before upgrading:

grep -rn 'RNEAT_generated_\|RNEAT_chimeric_' your_scripts/

If you must rewrite existing files instead, the recipes are in the README's "Upgrading from 2.0.0" section. Note BAM QNAMEs are covered by no in-tree tooling — filter-reads reads FASTQ, and the converter is bcftools-based — so for BAMs a sed through samtools view is the whole story.

---

New: subclonal heterogeneity for gen-cancer-reads (#405)

A tumor is no longer a single global purity scalar collapsing every somatic variant to ~one effective VAF. It is now a mixture of subclones at distinct cancer-cell fractions (CCF), with each somatic variant's observed alt fraction composing as purity × dosage × CCF — three orthogonal axes (normal contamination, per-copy multiplicity, subclonal prevalence). A het somatic at CCF f lands at ~f/2.

Three ways to specify the architecture:

• Inline — subclones:, a list of {ccf, weight}
• Fitted from real data — subclones_file: accepts a PyClone-VI per-mutation table or a PCAWG-11 / DPClust cluster table
• Replayed — somatic_vcf: honors a real tumor's observed VAF per site

Ground truth is emitted per somatic record as INFO/EIDOLON_CCF (intended CCF) and INFO/EIDOLON_VAF (intended observed VAF). Use EIDOLON_VAF, not FORMAT/AF, when scoring a somatic caller against merged output — FORMAT/AF is measured per-pass (tumor-only, = dosage × CCF), while EIDOLON_VAF is that × purity.

Validated on real data (NCSA Delta): a generative run reproduces a designed 3-cluster spectrum unbiased (mean error −0.003, MAE 0.026), and a reproductive replay of SEQC2 HCC1395 somatic SNVs over GRCh38 chr1 reproduces that tumor's full empirical VAF distribution (2,948 sites, all deciles) at Pearson r 0.99. See the caveat under Known issues.

---

Versioning and the public API

Releases before v3.0.0 were versioned by feel — v1.11.0 and v1.12.0 both changed the FASTQ read-name format in minor bumps, and v1.6.0 removed config fields under a bolded "Breaking change." A minor bump says "safe to upgrade without reading anything," which wasn't true of any of them.

SemVer requires a project to declare what its public API is, so eidolon's is now stated in the README and CHANGELOG:

Public API — a change here means a MAJOR bump

• Names and semantics of emitted VCF INFO tags, the VCF sample-column name, and emitted FILTER/FORMAT fields
• FASTQ/BAM read-name (QNAME) format, including the positional fields encoded after the prefix
• CLI subcommand names, flags, and configuration-YAML keys
• Model-file compatibility across versions

Not public API — may change in a MINOR or PATCH release

• The eidolon-core Rust library surface
• Log output and human-readable messages
• The exact simulated content for a given seed — reads are a random draw, so sampler and model changes legitimately alter output while preserving format

For a read simulator the emitted format is what downstream code actually binds to, which is why it heads the list — and why this release is major.

---

Fixes

Breakend (BND) truth records were unmatchable

De-novo BNDs were emitted as a single breakend with no ID, no MATEID, and a hardcoded N in the ALT that contradicted the real REF base. VCF 4.2 §5.4 represents an adjacency as a mate pair, and every SV caller emits pairs — so no comparison tool could match eidolon's BND truth, and BND recall read as 0% regardless of whether the reads were correct. (They were: junction-spanning chimeric reads were being generated correctly the whole time.)

Now emits both breakends with reciprocal ALTs, deterministic cross-referencing MATEIDs, and each record's own reference base:

CM000851.4  20592138  eidolon_bnd_..._1  T  T]CM000851.4:49516423]  SVTYPE=BND;MATEID=..._2
CM000851.4  49516423  eidolon_bnd_..._2  g  g]CM000851.4:20592138]  SVTYPE=BND;MATEID=..._1

This is an output change: a BND-containing truth VCF now carries two records per junction, so anything counting SV records will see roughly twice as many BND rows.

Three related fixes landed with it:

• A spec-correct BND mate pair double-counted the junction, emitting 2× chimeric reads at every breakpoint. This already affected anyone supplying paired BNDs via input_vcf: — the documented way to supply them.
• The golden VCF's ID column was hardcoded to ., discarding IDs supplied by an input VCF and leaving every MATEID dangling.
• MATEID is now declared in the header. An undeclared INFO tag streams fine VCF→VCF but hard-fails BCF translation — which bcftools concat | sort does internally.

Round-tripping a golden VCF

Feeding an eidolon golden VCF back in via input_vcf: or somatic_vcf: reproduced its routing tokens verbatim, producing duplicate EIDOLON_PROVENANCE keys with contradictory values and an undeclared EIDOLON_ORIGIN. Worse, the stale EIDOLON_PROVENANCE=denovo made the cancer origin-merge label a variant present in both passes as somatic — putting a germline carry-through into the somatic ground truth every benchmark scores against. write_vcf now strips its own routing tokens on passthrough while preserving EIDOLON_CCF/EIDOLON_VAF ground truth.

filter-reads on legacy FASTQ

An unrecognized read-name prefix silently dropped every record, producing an empty output with exit code 0 and a "completed successfully" log. Both prefixes are now accepted, with a one-time warning on the old one.

Validation harnesses (scripts/delta/, tools/)

If you use the bundled HPC validation harnesses:

• Three sites counted a filtered truth set, printed WARNING on zero, and scored against it anyway — reporting every caller at 0 recall as though the caller had failed. Now fatal, with actionable messages.
• The hap.py/som.py chr-prefixing workaround (written for Ensembl-numeric contigs) corrupted accession-named references such as CM000851.4 — any NCBI GCA assembly — failing with an opaque Error running BCFTOOLS … vcfcheck. Now applied only when som.py will actually rename the VCF's contigs.
• Reusing an output directory across a rebuild silently mixed artifacts from two code versions. Output directories are now version-stamped, and a mismatch regenerates rather than scoring new calls against an old truth.
• Two jobs sharing an output directory interleaved their writes and still produced numbers. Output directories and the shared bwa-mem2 index are now locked.

Testing

Adds vcf_validation.rs, which validates the golden VCF as a whole artifact — INFO/ALT declarations against usage, REF against the reference FASTA, BND ALT base against REF, MATEID resolution and reciprocity, per-contig sort order, column arity, ##contig presence. Every prior VCF test asserted that expected content was present; none asserted the file was well formed, which is how the defects above shipped green.

---

Known issues

Disclosed rather than discovered later...