Skip to content

4.0.0 2026-11-09

Latest
Latest

Choose a tag to compare

View all tags
@glennhickey glennhickey released this 09 Nov 15:16
· 10 commits to master since this release
v4.0.0
4c4d95b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Cactus 4.0.0 is available in the following forms:

WARNING: do not use the github automatically generated source files (Source code (zip) or Source code (tar.gz)), these are not correct.

The Docker images and binaries linked above are built using AVX2 extensions, and require a CPU that supports them, except the "Pre-compiled Binaries For Older CPU Architectures" which should be compatible with any 64-bit architecture (and, since version 2.3.1, support Cactus's pangenome pipeline).

Release Notes

This release makes Cactus's output reproducible and changes several defaults in both Progressive Cactus and Minigraph-Cactus. It is about 10% more accurate, at least 2-5x faster and uses less memory. It also adds output validation, per-job walltime estimates on Slurm, the ability to extend an existing pangenome and reintroduces support for multifurcations in guide trees.

General

  • A --configFile made from an older version's config should be remade from the new default config. It would get only some of the changes below, in combinations that are untested or worse: for example pecan without its new seed anchors, and on Slurm, lastz chunks a third of the size (slurmChunkScale is no longer read).
  • Output is now reproducible: the same input, command and binaries give the same result. cactus_consolidated used to give a slightly different alignment on every run, the order of the lastz chunks could depend on the filesystem, and minigraph graph construction depended on the order of the contigs in each input FASTA.
  • --validate (in cactus, cactus-align and cactus-pangenome) checks the output HAL base for base against the input sequences, and the new cactus-validate tool does the same for an existing HAL. cactus-prepare adds --validate to its cactus-align commands unless given --noValidate.
  • Cactus now estimates a walltime for every job, so on Slurm short jobs can go to short partitions and --slurmTime is no longer needed (it still overrides the estimates). --walltimeFactor, --minWalltime and --maxWalltime adjust them.
  • --doubleMem and --doubleTime are now on by default: a job killed for running out of memory or time is retried with twice as much.
  • cactus_consolidated is faster and uses less memory: CAF is optimized, sequences are held in half the memory, and BAR no longer uses nested parallelism. It also keeps the memory pages it frees when the job has room for them, which is much faster but takes 2-3x the peak memory (--consRetainPages). Its memory estimate was refit, with parameters in <consolidated> in the config in place of the <consolidatedMemory> table.
  • lastz is up to twice as fast, with the same output. Its jobs now use 90 Mbp chunks on every batch system (as Slurm runs already did, through slurmChunkScale, which is gone), with --queryhspbest=300000 to match: at 100000, chunks of that size dropped some real alignments.
  • Red is faster and uses less memory, and is now run with -cap 99.8 -sketch 1 on every contig (the prefilter that set small contigs aside is gone). These mask high-copy sequence that Red alone leaves, which makes assemblies with little or no repeat masking, such as NCBI's WindowMasker-only ones, much cheaper to align. redOpts="" in the config restores Red alone.
  • Indels in repeats are placed more consistently, rather than by the orientation of the input sequences: to the left on the reference in pangenomes (<graphmap leftAlign>, <bar poaOrientation>), and canonically in the lastz alignments of progressive runs (<blast leftAlign>).
  • Fix jobs that changed their input files when the job store and work directory share a filesystem: cactus-halAppendSubtrees appended to the HAL it was given, cactus-graphmap-join removed _MINIGRAPH_ from the HALs given to --hal, and a retried job could start from a half-written file.
  • Fix bundled tools that were built without compiler optimization: vcflib (vcfwave), odgi and phast, and, in the binary release, samtools, bcftools and HDF5.
  • --hdf5Codec added to cactus-graphmap-join and cactus-pangenome, and fixed in cactus-prepare, where it never reached halAppendSubtree.
  • Components with restrictive licenses are replaced or removed: Blossom V by an MIT implementation of Edmonds' matching algorithm (<reference matchingAlgorithm="edmonds">, which blossom5 is now an alias for), GNU libavl in sonLib and the 3-edge-connected components code in pinchesAndCacti by MIT code, and LAST is no longer used.
  • x86 binaries are now built for x86-64-v3 (was -mavx2), bundled tools included. CACTUS_NATIVE_BUILD=1 builds for the local CPU instead, and CACTUS_ARCH_FLAGS sets the flags directly (see DEVELOPMENT.md).
  • Minigraph-Cactus can now be built on Mac.
  • Update to Toil 9.6.0, jemalloc 5.3.1, and the latest FastGA and hal (which copies DNA faster, speeding up halAppendSubtree and halMergeChroms).
  • Setting CACTUS_BAR_DUMP_DIR dumps every window that BAR gives to abPOA or minipoa, with a command line that replays it.

Progressive Cactus

  • pecan is the default base aligner in BAR again (thanks @ofirr) (<bar baseAligner="pecan">; pangenomes keep abPOA through <bar pangenomeBaseAligner>). It is now faster and deterministic, matches abPOA on closely related genomes, is more accurate on distant ones, and uses less memory. baseAligner="abpoa" restores the previous default. baseAligner replaces <bar partialOrderAlignment>, which is no longer read.
  • minipoa added as a third base aligner (<bar baseAligner="minipoa">, x86 builds only): as accurate as abPOA in our tests, with less memory.
  • Multifurcations (ancestors with more than two children) are allowed again, and the allow_multifurcations config option is gone. They were refused since v2.7.2 because groups of children could be left unaligned to each other; each pair of children is now also chained on its own to join them (<blast multifurcationPairSecondaries>).
  • Fix paffy trim never trimming the end of forward-strand alignments (<blast pafTrimIdentity> is raised from 0.2 to 0.35 to go with it), and paffy tile and paffy to_bed miscounting the bases of reverse-strand alignments with insertions.
  • Fix three non-default options that did not work: <blast outputSecondaryAlignments="1"> did not give CAF the secondary alignments from paffy tile, the CAF alignment filters that sort their input (<caf alignmentFilter="singleCopy"> and similar) aligned nothing, and <blast mapper="minimap2"> took primary and secondary status from minimap2's tags rather than from the chaining.
  • Ancestral bases are now called with the input tree's branch lengths (times --branchScale), without the lengthening of ancestral branches that upweightAncestorDistances applies for alignment.
  • --coalescenceLimit added to cactus-hal2chains, to include homology through paralogies that coalesce above the MRCA of the query and target.
  • Fix cactus_consolidated aborting in BAR on an unaligned stretch of more than 2^31 bases, as on chromosomes over 2 Gb.
  • Fix garbled commands in cactus-update-prepare plans, and the HAL merging commands left in add branch plans.

Minigraph-Cactus

  • Genome names now follow PanSN: a haplotype is SAMPLE#HAPLOTYPE (HG002#1), and names are used as they are throughout, in the HAL and MAF as well as in the graph and VCF. The old SAMPLE.HAPLOTYPE form is still read in the seqfile, with a warning (HG002.1 becomes HG002#1), so a sample name that ends in .N must be given with its haplotype to be kept (GCA_054883195.1#0). dotHaplotypes="0" in <graphmap> turns that reading off, and will become the default. See "Sample Names" in doc/pangenome.md.
  • minigraph is now built from a fork, pinned at 6ef255e. It is faster (it uses several threads per contig), places copy-number variants by their own sequence rather than through a near-identical paralog (--ggs-cnv, on by default in minigraphConstructOptions), and builds the same graph whatever the order of the input contigs. It also fixes a bug, in every Cactus release since v2.1.0, that reversed the CIGARs of some GAF records: few of those mappings reached cactus-align, but GAFs from earlier versions should not be reused with --inGAF.
  • Scoring models are now trained by default (<graphmap lastTrain>; --lastTrain is deprecated), with cPecan's pair HMM instead of LAST's last-train. Training can no longer fail the run, and the model now also sets minigraph's base-alignment penalties when mapping (<graphmap lastTrainMap>). --scoresFile still takes a last-train file.
  • minigraph graphs are now cleaned up with rgfa-zip after construction (zipAlleles in <graphmap>, on by default): an allele that minigraph stored as novel sequence but that copies the reference it bypasses, inversions included, is merged into it. The graph as minigraph built it is kept beside the new one, as .sv.unzipped.gfa.gz. See "Zipping Alleles onto the Reference" in doc/pangenome.md.
  • Overlapping minigraph mappings are now resolved per query segment instead of by deleting whole mappings, with identity deciding before MAPQ and length (GAFOverlapFilterExact and GAFOverlapFilterExactWinIdent in <graphmap>, both on by default), which anchors more sequence. See "Overlapping Mappings" in doc/pangenome.md.
  • cactus-pangenome --mgSplit (and cactus-panpatch --mgSplit) now builds each chromosome's graph against the whole primary reference genome, so that off-chromosome mappings compete as they do in the whole-genome pipeline (wholeGenomeRef="0" in <graphmap_split> gives the v3.3.0 behaviour, which the step-by-step commands still have). Its reference-only first pass now writes its graph and mappings as <outName>.refonly.*.
  • All pangenomes now chain with a minimum length of 256 (default_branch_len_pangenome in <decomposition>). --mgSplit pangenomes already did, by accident; default ones used 2048.
  • cactus-pangenome --inGFA (also cactus-minigraph --inGFA) starts from an existing minigraph graph, to add genomes to a pangenome without rebuilding it or to rerun the later stages, and --inGAF (also cactus-graphmap --inGAF) reuses its mappings, which needs zipAlleles="0" when adding genomes. Not supported with --mgSplit or --collapse. See "Starting From an Existing Graph or Mappings" in doc/pangenome.md.
  • --minIdentity added to cactus-pangenome, cactus-graphmap and cactus-align. cactus-graphmap --outputGAFDir, which did nothing, is removed.
  • Fix cactus-align filtering out some of the reference's own mappings, which cactus-graphmap-split exempts from its MAPQ, length and identity thresholds.
  • The GAF that cactus-graphmap and cactus-pangenome write now uses PanSN names, like the minigraph GFA it goes with.
  • The clip graph is cleaner. Edges that span more than 10 Mb of the reference are now removed by default, along with whatever that leaves dangling (delEdgeFilter in <graphmap_join>; --delEdgeFilter 0 turns it off). The tangled fringe that aligners leave around clipped regions such as satellites is trimmed (--clipFlank, 50 kb by default). And two makers of giant single-contig snarls are filtered: deletions that only one contig's mappings imply (delFilterMaxRemove in <graphmap>), and inversions that clipping left attached at one end only (clipInversionMin in <graphmap_join>).
  • Chromosome ends whose subtelomere differs from the reference's are now kept, along with their telomeres. The last 100-500 kb of a chromosome is often a block shared with other chromosome ends, and in the HPRC v2.1 graph 81% of the 6p ends lost such a block to clipping (68% of 8p, 42% of 9q and 28% of 16p), while most ends lost 2-20 kb of telomere. minigraph's new --ggs-ends (in minigraphConstructOptions, with --ggs-end-min 10000) adds the part of a contig that overhangs a chromosome end of the graph as a tip of its own. telomereAnchors in <graphmap> aligns every contig's telomere array to a synthetic telomere for an arm of its reference contig (tel_<contig>p or tel_<contig>q, in the _MINIGRAPH_ genome), as constraints that CAF keeps whatever its minimum chain length, so that telomeres align across haplotypes and clipping keeps them and the tips. The two go together, and both are on by default. On 40 HPRC haplotypes of 8 chromosomes, the clip graph kept 99.9% of the telomere sequence (39% without them) and 98% of the subtelomere blocks from other chromosome ends (none without them), for 0.26% more sequence, and vg haplotypes could sample 37-40 of the 40 haplotypes at every chromosome end (2-38 without them). Each alignment job picks its telomere motif from telomereMotifs, whose default covers vertebrates, plants, most insects, nematodes and others: on 13 Arabidopsis thaliana genomes, all 104 telomeres were anchored. PAF lines tagged ct:i:1 given to cactus-align now go to cactus_consolidated as constraints. The anchors need edlib, a new Python dependency. See Subtelomeres and Telomeres.
  • Fix --haplo full failing in vg haplotypes with top-level chain N is a loop when an end of the reference lies on a cycle. The reference's paths now end in tips in the full graph (fullStubbify in <graphmap_join>).
  • --gref with --haplo now writes a .gref.hapl, as the base graph's .hapl does not work with gref.gbz.
  • Update vg to a build of its master branch (453e6c0, past v1.78.0), which has fixes to vg clip that the join relies on. The .hapl files from --haplo are now in vg's version 7 format, which vg v1.78.0 and older cannot read.
  • The giraffe and haplotype index jobs now reserve memory by the size of the graph's snarls rather than at most 128 GiB, which large graphs outgrew, and --indexMemory now caps every job in the join.
  • --pgrealign (experimental) realigns tandem repeats with pgrealign.
  • filterAC0 now also trims the alleles that no sample carries from the VCF sites it keeps.
  • hal2vg's conversion now needs about half the memory (thanks @yeeus). hal2vg and its tools are now built from source by downloadPangenomeTools, which adds the Boost headers (libboost-dev) to the build dependencies.
  • cactus-panpatch: --assemblyErrorBeds steers patching away from suspected assembly errors, panpatch-summary.tsv summarizes every sample of a run, and telomeres are now completed by default.

Contributors

ofirr and yeeus
Assets 5
Loading