Skip to content

wgsextract-cli v0.6.4

Choose a tag to compare

View all tags
@github-actions github-actions released this 01 Nov 16:54
· 822 commits to main since this release
v0.6.4
284bfd2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

New

  • Streamed paired realignment: eligible fresh paired BAM/CRAM inputs feed
    FASTQs directly into BWA or BWA-MEM2 instead of writing and rereading gzipped
    FASTQs. Pair filtering, fixed batches, duplicate marking and decoded results
    stay unchanged. Eligibility is bounded by input size and available memory;
    30× whole genomes still use the existing FASTQ-file path, as do cached
    or ineligible runs. This is not a whole-genome streaming speed claim. (#257)
  • Native Windows GATK acceleration: the pinned GKL sidecar supplies native
    AVX/OpenMP PairHMM and AVX2 Smith-Waterman for GATK 4.6.2.0, without WSL.
    Released-library numerical and tiny HaplotypeCaller parity checks reject
    silent Java fallback. Optional GATK dependencies must already be installed.
    (#227, #228, #229)
  • More microarray exports:
    • New, separately selectable MyHeritage GSA and Ancestry marker layouts;
      older layout IDs are retained. (#210)
    • --formats aadr_1240k,aadr_ho,aadr_1240k+ho writes pinned AADR research
      panels in PLINK and EIGENSTRAT formats, with documented allele conventions.
      (#211)
    • Opt-in --formats aadr_2m exports the official AADR v66.p1 2M panel
      (2,142,271 sites) as PLINK and EIGENSTRAT. Install its checksum-pinned
      metadata explicitly with ref aadr --install --ref LIBRARY; extraction
      never downloads it, and all does not include it. GRCh38 requires an
      installed forward chain; 2M CHM13/T2T projection is not supported.
      AADR metadata is CC0 1.0; attribution to the Reich Lab, Harvard Dataverse
      release 14.0 and Mallick et al. (2024) is retained. (#280, #270)
    • Native CHM13v2 calling projects to GRCh37/rCRS exports using the exact
      original-mitochondrial, masked-Y reference. Other CHM13 reference variants
      are rejected; ambiguous mappings remain no-calls. (#219)
    • Optional --indel-manifest FORMAT=PATH calls verified, manifest-defined
      insertion/deletion probes, including supported CHM13 alignment input.
      It does not infer probe orientation or enable unrestricted indel discovery.
      Missing evidence and unsupported probes remain no-calls. (#217, #226)
  • Optional microarray quality and identity policies: sample GQ/DP thresholds,
    FILTER exclusion, explicit absent-site hom-ref policy and --call-reasons
    diagnostics; --strand-ambiguous no-call conservatively masks known A/T and
    C/G sites, and --resolve-rsids checks merged/withdrawn marker identities.
    Existing defaults and vendor schemas are unchanged. (#212, #214)
  • Targeted Y analysis: extract ydna-bam --poz selects licensed GRCh37/38
    Poz Y intervals; extract ydna-vcf --ybrowse calls installed, verified
    GRCh38 yBrowse sites and adds their lineage annotations. Both are opt-in;
    yBrowse data requires explicit setup and is never downloaded during calling.
    (#249, #258)
  • Offline QC and coverage: qc multiqc --input REPORT_OR_DIRECTORY
    aggregates FastQC reports with an already-installed MultiQC, without online
    services or opening a browser. info calculate-coverage --bins streams
    per-chromosome counts and fractions at depth 0 / 1–3 / 4–7 / 8+, without a
    per-base output file. (#265, #247)
  • Long-run controls: stderr progress heartbeats and optional --log-file
    persistent logs; --tmp-dir DIRECTORY routes sorting, GATK and realign
    staging to a chosen writable volume; opt-in --keep-awake requests sleep
    inhibition for the command's lifetime. (#251, #254, #248)
  • Optional Ultima GATK interval scattering: vcf gatk --scatter bounds
    independent flow-aware workers by threads and available memory, then gathers
    successful intervals in reference order. Serial remains the default; GPU and
    non-flow inputs are rejected. (#208)
  • Extraction and lineage conveniences: extract unmapped --fastq preserves
    pairs, single-end reads and orphan mates in separate outputs;
    lineage y-haplogroup --tree-links and lineage mt-haplogroup --tree-links
    add opt-in lineage-tree links. (#246)
  • Platform preview: native Windows ARM64 qualification supplies opt-in
    packaging and native-only runtime checks, with no Prism, WSL or x64
    substitution. Qualified BWA/BWA-MEM2 and bcftools paths are separate from the
    unchanged Windows x64 installation. ARM64 packages are not yet included
    in published release artifacts or the default lock    ; ARM64 GATK/GKL and
    DeepVariant remain unsupported in this phase. (#272)

Fixes

  • Full-human BWA-MEM2 indexing on native Windows: 64-bit reference lengths
    and bounded native I/O fix the index truncation above 4 GiB. The installer
    now pins checksum-verified bwa-mem2 2.3-wgse_2 from windows-tools-8;
    the temporary large-reference guard is removed only after native full-human
    indexing was verified. Windows builds its own index—no Linux-built index
    substitution is needed. (#264, #268, #275)
  • Resumable, verified downloads: shared HTTP handling extends resumable
    partial files and checksum-verified promotion/cache reuse to benchmark and
    example downloads. Invalid SHA-256 declarations fail before destination
    changes, and failed MD5 checks cannot poison the shared cache. Installer
    Pixi downloads are pinned and verified before execution; catalog reference
    sequence signatures are checked after indexing. (#260, #252)
  • Safer recovery and outputs: interrupted reference indexes are detected
    and rebuilt through staged, locked publication; malformed BAM/CRAM inputs
    fail early. Opt-in --atomic-output preserves an existing result when its
    replacement fails, without changing default provenance headers. (#256)
  • Windows cancellation: Ctrl+C remains responsive while GATK runs, and
    launcher cleanup terminates registered Java descendants instead of leaving
    them running after cancellation. (#263, #269)
  • Dependency preflight: optional environments are validated, not installed
    at runtime; missing dependencies report the explicit locked install command.
    Unsupported Linux ARM64 default installations fail early rather than
    attempting an unusable environment. (#245, #255)
  • Clearer resource and reference advice: Ultima BWA memory estimates account
    for read length, thread count and Linux cgroup limits without silently changing
    requested threads. Reference errors name the matching library install command;
    VEP uses canonical --assembly wording while retaining its deprecated alias.
    Realign no longer prints misleading fixed per-stage hour estimates. (#224,
    #246, #260, #238)

Validated: Linux + native Windows small suites; chr20 BWA on both and
BWA-MEM2 on Linux: decoded alignments/SNPs/indels EXACT versus v0.6.3, streaming
confirmed; native Windows full-human index 5/5 Linux goldens EXACT.
Scientific gate 0cac0eb and native x64 small-suite delta gate a7bdb2f.

Known issues

  • DeepVariant can fail hours into a run on CRAM 3.1 inputs with
    Failure to decode slice / DATA_LOSS: Failed to parse SAM record. The
    CRAM is valid; DeepVariant images (including 1.10.0) embed htslib 1.18,
    whose SSE4 rANS decoder has an upstream bug (samtools/htscodecs#102, fixed
    in htslib 1.19). CRAM 3.1 is the default for CRAMs written by samtools 1.22+,
    including wgsextract's. Workaround: convert the input to CRAM 3.0 first,
    e.g. samtools view -C --output-fmt-option version=3.0 -T REF -o in.v30.cram in.cram && samtools index in.v30.cram,
    or use BAM. An automatic fix is planned for v0.6.5. (#283)

Full changes: v0.6.3...v0.6.4

Assets 2
Loading