v3.0.0

eidolon v3.0.0

Two headline changes: intra-tumor heterogeneity (a tumor is now a mixture of subclones, not a single purity scalar), and a breaking rename of every token eidolon emits — completing the rusty-neat → eidolon rename begun in v2.0.0.

From this release eidolon follows Semantic Versioning 2.0.0 and declares a public API, so future version numbers mean something specific.

⚠️ If you have scripts that parse eidolon's VCFs, FASTQs or BAMs, read "Upgrading" before installing. Nothing about the reads or variants changed — only the names of emitted tokens. But a parser matching the old names will match nothing, silently, with exit code 0.

---

Upgrading from 2.0.0

What changed

Surface	v2.0.0 and earlier	v3.0.0	Migration
VCF INFO tags	NEAT_ORIGIN, NEAT_PROVENANCE, NEAT_REASON, NEAT_CCF, NEAT_VAF	EIDOLON_*	converter provided
VCF sample column	NEAT_simulated_sample	EIDOLON_simulated_sample	converter provided
FASTQ read names	@RNEAT_generated_*, RNEAT_chimeric_*	@EIDOLON_generated_*, EIDOLON_chimeric_*	not converted — see below
BAM read names (QNAME)	RNEAT_generated_*	EIDOLON_generated_*	not converted — see below

VCFs — convert them

tools/migrate_legacy_tokens.sh old_truth.vcf.gz new_truth.vcf.gz
tools/migrate_legacy_tokens.sh --check old_truth.vcf.gz    # report only; exit 10 = legacy tokens present

Idempotent, so it's safe to call unconditionally in a pipeline — it's a no-op on a file that's already current or isn't an eidolon VCF. It rewrites the header declarations and the record fields together, so the result is a valid VCF.

You cannot work around this with a dual-name bcftools filter. bcftools rejects an undefined tag when it parses the expression, so even

-i 'INFO/EIDOLON_ORIGIN="somatic" || INFO/NEAT_ORIGIN="somatic"'

fails outright on a file carrying only one of the two. Convert the file instead.

Read names — make your parser prefix-agnostic

Read names are not rewritten, in FASTQ or BAM. Rewriting files that are routinely hundreds of GB to change ~19 bytes per record isn't a reasonable ask, so eidolon filter-reads accepts both prefixes natively and warns once when it sees an old one.

Only the prefix changed. Everything after _generated_ — <contig>_<start>_<end>_<uniq>/<mate> — is byte-for-byte identical between versions. So rather than rewriting files, strip the prefix and your script works against either version, and against any future rename:

samtools view x.bam | cut -f1 | sed -E 's/^[A-Z]+_generated_//'
zcat x_r1.fastq.gz  | awk 'NR%4==1' | sed -E 's/^@[A-Z]+_generated_//'

Same for chimeric reads: sed -E 's/^@?[A-Z]+_chimeric_//'.

Audit before upgrading:

grep -rn 'RNEAT_generated_\|RNEAT_chimeric_' your_scripts/

If you must rewrite existing files instead, the recipes are in the README's "Upgrading from 2.0.0" section. Note BAM QNAMEs are covered by no in-tree tooling — filter-reads reads FASTQ, and the converter is bcftools-based — so for BAMs a sed through samtools view is the whole story.

---

New: subclonal heterogeneity for gen-cancer-reads (#405)

A tumor is no longer a single global purity scalar collapsing every somatic variant to ~one effective VAF. It is now a mixture of subclones at distinct cancer-cell fractions (CCF), with each somatic variant's observed alt fraction composing as purity × dosage × CCF — three orthogonal axes (normal contamination, per-copy multiplicity, subclonal prevalence). A het somatic at CCF f lands at ~f/2.

Three ways to specify the architecture:

• Inline — subclones:, a list of {ccf, weight}
• Fitted from real data — subclones_file: accepts a PyClone-VI per-mutation table or a PCAWG-11 / DPClust cluster table
• Replayed — somatic_vcf: honors a real tumor's observed VAF per site

Ground truth is emitted per somatic record as INFO/EIDOLON_CCF (intended CCF) and INFO/EIDOLON_VAF (intended observed VAF). Use EIDOLON_VAF, not FORMAT/AF, when scoring a somatic caller against merged output — FORMAT/AF is measured per-pass (tumor-only, = dosage × CCF), while EIDOLON_VAF is that × purity.

Validated on real data (NCSA Delta): a generative run reproduces a designed 3-cluster spectrum unbiased (mean error −0.003, MAE 0.026), and a reproductive replay of SEQC2 HCC1395 somatic SNVs over GRCh38 chr1 reproduces that tumor's full empirical VAF distribution (2,948 sites, all deciles) at Pearson r 0.99. See the caveat under Known issues.

---

Versioning and the public API

Releases before v3.0.0 were versioned by feel — v1.11.0 and v1.12.0 both changed the FASTQ read-name format in minor bumps, and v1.6.0 removed config fields under a bolded "Breaking change." A minor bump says "safe to upgrade without reading anything," which wasn't true of any of them.

SemVer requires a project to declare what its public API is, so eidolon's is now stated in the README and CHANGELOG:

Public API — a change here means a MAJOR bump

• Names and semantics of emitted VCF INFO tags, the VCF sample-column name, and emitted FILTER/FORMAT fields
• FASTQ/BAM read-name (QNAME) format, including the positional fields encoded after the prefix
• CLI subcommand names, flags, and configuration-YAML keys
• Model-file compatibility across versions

Not public API — may change in a MINOR or PATCH release

• The eidolon-core Rust library surface
• Log output and human-readable messages
• The exact simulated content for a given seed — reads are a random draw, so sampler and model changes legitimately alter output while preserving format

For a read simulator the emitted format is what downstream code actually binds to, which is why it heads the list — and why this release is major.

---

Fixes

Breakend (BND) truth records were unmatchable

De-novo BNDs were emitted as a single breakend with no ID, no MATEID, and a hardcoded N in the ALT that contradicted the real REF base. VCF 4.2 §5.4 represents an adjacency as a mate pair, and every SV caller emits pairs — so no comparison tool could match eidolon's BND truth, and BND recall read as 0% regardless of whether the reads were correct. (They were: junction-spanning chimeric reads were being generated correctly the whole time.)

Now emits both breakends with reciprocal ALTs, deterministic cross-referencing MATEIDs, and each record's own reference base:

CM000851.4  20592138  eidolon_bnd_..._1  T  T]CM000851.4:49516423]  SVTYPE=BND;MATEID=..._2
CM000851.4  49516423  eidolon_bnd_..._2  g  g]CM000851.4:20592138]  SVTYPE=BND;MATEID=..._1

This is an output change: a BND-containing truth VCF now carries two records per junction, so anything counting SV records will see roughly twice as many BND rows.

Three related fixes landed with it:

• A spec-correct BND mate pair double-counted the junction, emitting 2× chimeric reads at every breakpoint. This already affected anyone supplying paired BNDs via input_vcf: — the documented way to supply them.
• The golden VCF's ID column was hardcoded to ., discarding IDs supplied by an input VCF and leaving every MATEID dangling.
• MATEID is now declared in the header. An undeclared INFO tag streams fine VCF→VCF but hard-fails BCF translation — which bcftools concat | sort does internally.

Round-tripping a golden VCF

Feeding an eidolon golden VCF back in via input_vcf: or somatic_vcf: reproduced its routing tokens verbatim, producing duplicate EIDOLON_PROVENANCE keys with contradictory values and an undeclared EIDOLON_ORIGIN. Worse, the stale EIDOLON_PROVENANCE=denovo made the cancer origin-merge label a variant present in both passes as somatic — putting a germline carry-through into the somatic ground truth every benchmark scores against. write_vcf now strips its own routing tokens on passthrough while preserving EIDOLON_CCF/EIDOLON_VAF ground truth.

filter-reads on legacy FASTQ

An unrecognized read-name prefix silently dropped every record, producing an empty output with exit code 0 and a "completed successfully" log. Both prefixes are now accepted, with a one-time warning on the old one.

Validation harnesses (scripts/delta/, tools/)

If you use the bundled HPC validation harnesses:

• Three sites counted a filtered truth set, printed WARNING on zero, and scored against it anyway — reporting every caller at 0 recall as though the caller had failed. Now fatal, with actionable messages.
• The hap.py/som.py chr-prefixing workaround (written for Ensembl-numeric contigs) corrupted accession-named references such as CM000851.4 — any NCBI GCA assembly — failing with an opaque Error running BCFTOOLS … vcfcheck. Now applied only when som.py will actually rename the VCF's contigs.
• Reusing an output directory across a rebuild silently mixed artifacts from two code versions. Output directories are now version-stamped, and a mismatch regenerates rather than scoring new calls against an old truth.
• Two jobs sharing an output directory interleaved their writes and still produced numbers. Output directories and the shared bwa-mem2 index are now locked.

Testing

Adds vcf_validation.rs, which validates the golden VCF as a whole artifact — INFO/ALT declarations against usage, REF against the reference FASTA, BND ALT base against REF, MATEID resolution and reciprocity, per-contig sort order, column arity, ##contig presence. Every prior VCF test asserted that expected content was present; none asserted the file was well formed, which is how the defects above shipped green.

---

Known issues

Disclosed rather than discovered later. None are regressions in this release; all predate it.

• #450 — the subclonal-VAF validation harness silently excludes the lowest-VAF sites it exists to test (bcftools call -m drops a hom-ref call's ALT and its read count). The generative validation numbers above are therefore conditional on the clusters that were scored; the reproductive HCC1395 run is far less affected (91.5% of sites scored). This is a measurement limitation, not evidence the simulator is wrong.
• #457 — truvari bench cannot match sizeless breakends, so BND recall from the SV harness reads 0 even with correct truth. Per-type precision from that harness is also invalid (the query isn't type-filtered); per-type recall is sound.
• #458 — BND realism: de-novo breakends are intra-chromosomal only, and only 1 of the 4 VCF adjacency orientations is emitted.
• #444 — compare-vcfs writes FN_with_reasons.vcf / FP.vcf with a minimal header while passing input INFO through, so those debugging artifacts carry undeclared tags and fail BCF translation.

Install

Binaries are attached to this release. Bioconda (conda install -c bioconda eidolon) follows shortly — the recipe bump needs this release's tarball checksum.

Full detail, including the reasoning behind each decision, is in CHANGELOG.md.