wgsextract-cli v0.6.3

WGS Extract CLI v0.6.3

Native Windows full-genome validation, reproducibility and memory improvements, and several Windows and download fixes.

New

• Optional parallel SNP/indel calling: vcf snp --parallel and
  vcf indel --parallel process indexed whole-genome alignments with up to
  --threads contig workers, resolve sex/ploidy once, and concatenate in
  reference-index order. Serial calling remains the default. bcftools uses
  a random subsampling stream per process at sites with more than 255 bases;
  starting independent contig workers changes those streams and can change
  likelihoods, INFO values, or low-quality calls. Full-genome testing found
  nine differing sites among roughly 4.4 million SNPs, despite passing smaller
  comparisons. Parallel output is therefore not guaranteed byte-identical
  to serial output; opt in only if that tradeoff is acceptable. Explicit
  regions, one thread, and unsupported input/index layouts use one stream.
  (#175, #199; exactness investigation #197)
• Thread-independent BWA batches: BWA and BWA-MEM2 now use fixed
  -K 100000000 batches in short-read, Ultima, and pet alignment. This removes
  thread-count-dependent batching; one-million-pair tests produced identical
  decoded alignment bodies at two, four, and six threads. The Windows MEM2
  subset gate recorded a 3.2% throughput loss, specifically accepted for
  reproducibility; this is not a full-genome performance claim. (#176)
• Measured alignment memory preflight: account for resident reference
  indexes, aligner working memory, duplicate marking, threaded sorting, and
  overhead against currently available memory on Linux, macOS, and Windows.
  Automatic selection falls back to BWA when MEM2 cannot fit; explicit MEM2
  refuses with advice to select BWA or lower --memory. BWA warns only.
  Read-dependent components and sort spill now scale with input read count,
  using bounded compressed-FASTQ sampling or indexed BAM/CRAM counts rather
  than reference length. The measured H2 input estimates about 18 GiB for BWA
  and 29 GiB for MEM2 at six threads / 256M sort memory. (#179, #194)
• Lower BWA-MEM2 sorting memory: cap samtools sort at 256 MiB per thread,
  preserve smaller explicit budgets, and use that effective budget throughout
  aligner selection, estimates and safety guards. BWA and other aligners are
  unchanged. Matched five-million-pair chr20-reference trials were strictly
  exact on Linux and Windows, with no observed throughput regression. (#178)
• Advisory memory and disk estimates before CPU alignment/realignment,
  BAM/CRAM sorting/conversion, and SNP/indel/GATK calling: compact
  OK / LOW / UNKNOWN rows compare estimated demand with available
  resources, combining concurrent demands on shared volumes. Estimates include
  output/index space, extracted FASTQs, retained Ultima/GATK BAM intermediates,
  and unresolved reference-storage uncertainty. Estimation failures warn and
  never introduce a new command refusal; existing safety guards remain.
  A strict, read-only, versioned JSON-compatible API is available for future
  GUI integration. (#198)

Fixes

• Indels at ambiguous reference bases no longer abort full hs38DH calling.
  Normalize IUPAC reference codes to concrete VCF-compatible bases (for example
  R to A) while retaining ordinary records unchanged. Consensus generation
  reconciles compatible ambiguity in temporary inputs without changing the
  exported VCF or its called ALT bases. (#190; fixes #191)
• Trio input selection now honors --input for the proband when
  --proband is absent, with shared --vcf-input precedence and intact
  BAM/CRAM support. Normalize parent/proband chromosome aliases beyond the
  primaries, including random, unplaced, and ALT contigs, even when a contig
  has no proband records. This fixes Nebula-style gVCF merging against
  chr-prefixed references and preserves annotation name round-trips.
  (#195; fixes #196)
• Native Windows Java/GATK paths: paths entered as R:/... are passed
  with native separators for references, inputs, outputs, interval files,
  temporary directories, and dictionaries, preventing
  ProviderNotFoundException: Provider "R" not found. Java launch paths for
  FastQC/HaploGrep are covered too; explicit URIs and POSIX argv are preserved.
  Hosted Windows CI includes a real regional GATK smoke test. (#201; fixes #200)
• Safe reference re-downloads: refreshes start at byte zero rather than
  promoting a completed destination into a resumable partial. Only a genuine
  .partial is resumed. curl, urllib, and segmented downloads preserve the
  previous destination until the candidate passes configured verification,
  then atomically replace it; failed verification leaves the old file intact.
  (#203; fixes #202)
• Correct GATK output warnings: indexes produced by GATK/htsjdk can report
  zero records even for nonempty VCFs. Confirm indexed zero counts with a
  bounded single-record probe before warning that no variants were called.
  Variant contents are unchanged. (#206; fixes #205)
• Clear early Windows refusals for unsupported commands, including
  pyBigWig-backed PhyloP/GERP, instead of failing after input preparation or
  creating output directories. Table-backed PhyloP remains available.
  Parabricks GPU modes explicitly require Linux with an NVIDIA GPU. (#174)

Native Windows validation

We validated a full 30× human genome on native Windows (no WSL or MSYS2) against Linux:

• Byte-identical to Linux: full-genome BWA alignment, serial SNP and indel calling, trio analysis, BAM/CRAM sorting and conversion, read extraction, annotation, lineage and QC.
• Command coverage: 43 benchmark and 22 supplemental native-Windows command cases passed, including reference downloads. Optional FreeBayes was not installed and so was not tested.
• GATK HaplotypeCaller calls the same sites and genotypes on Windows and Linux (124,581 chr20 records, 0 site or genotype differences). However, 26 records differ in QUAL/PL/GQ/QD/MQ/DP/AD, and Windows is roughly 4–5× slower. GATK's Intel GKL native kernels do not ship for Windows, so GATK falls back to pure-Java PairHMM and Smith-Waterman. Filters on confidence or depth may therefore behave slightly differently on Windows. Tracking: #207.
• Parallel SNP/indel calling (--parallel) is not byte-identical to serial calling on full genomes, which is why serial remains the default (#197, #199).
• Not validated in this release: full-human BWA-MEM2 alignment (needs about 48 GB RAM, #188) and full-genome realign on Windows (#189).

BWA-MEM2 sort-memory cap: measured impact

On matched 5-million-pair runs (6 vCPU, 10 GiB), Linux and Windows output was byte-identical with and without the cap. Wall time was unchanged (Windows 702 s → 700 s; Linux 716 s → 715 s), and peak memory fell by about 1.5 GiB on both. Windows ran within 2% of Linux.

Full Changelog: v0.6.2...v0.6.3