Skip to content

DiMA 1.0.0

Latest

Choose a tag to compare

@github-actions github-actions released this 07 Oct 12:38
· 1 commit to master since this release
6bc8f0e

First release of the Rust implementation of DiMA (Tharanga et al. 2025, Briefings in
Bioinformatics
26(1):bbae607, doi:10.1093/bib/bbae607).
It replaces every earlier version of this repository; results of those versions are not
comparable (see "Fixed").

Methodology

  • Entropy correction (--estimator dima, the default) follows the paper's Fig. 2 exactly:
    subsample sizes of R% of the sequences, with R from max(50, ⌈100·T/N⌉) rounded up to a
    multiple of 5 in steps of 5, plus 100%; sizes ⌈R·N/100⌉; sampling with replacement; the
    entropy regressed on 1/n; the intercept is the estimate, with the paper's fallbacks (raw
    entropy when N ≤ T or with fewer than 5 points). It is computed as the
    exact expectation of the random intercept, so results are reproducible bit for bit, with the
    random intercept's mean and no resampling noise (Rao 1945; Blackwell 1947,
    doi:10.1214/aoms/1177730497): the limit of
    averaging many random subsets, as DiMA's predecessors did (Khan et al. 2017,
    doi:10.1186/s12920-017-0301-2). The expected
    intercept is never negative, so the paper's negative-intercept rule never applies; DiMA
    v5.0.9 applied it to each random run, which raised its average at near-conserved positions
    (MERS-CoV spike protein eligible mean 0.126 bits here, about 0.142 averaged over v5.0.9 runs;
    HIV-1 pol unchanged). Every reported entropy is checked to be a finite number of at least 0
    bits; a violation is an internal error (exit 1), never replaced by another value. On the paper's HIV-1 pol dataset the eligible mean is 5.3504 bits (paper: ~5.35).
  • Alternative estimators: chao-wang-jost (Chao, Wang & Jost 2013,
    doi:10.1111/2041-210X.12108), miller-madow
    (Miller 1955) and raw (plug-in). Each position records the estimator or fallback used.
    The second term of Chao–Wang–Jost is evaluated exactly in a time independent of the number of
    sequences (hypergeometric series or Gauss's continued fraction: Hankin 2015,
    doi:10.32614/RJ-2015-022; Van Vleck 1901,
    doi:10.2307/1967627; Henrici & Pflüger 1966,
    doi:10.1007/BF02166031; Jones & Thron 1974,
    doi:10.1090/S0025-5718-1974-0373265-5;
    Alzer 1997, doi:10.1090/S0025-5718-97-00807-7),
    where summing it over every sequence would cost about 10% of a run and lose up to 8 digits.
  • Sample coverage per position (Chao & Jost 2012, Eq. 4a,
    doi:10.1890/11-1952.1), with an optional
    --min-coverage flag.
  • Positions without support (NS) have no entropy (null/NA), never 0 bits; summaries
    cover positions with support above T only, with the paper's averaging also reported as
    all_positions_summary.
  • Lowest entropy is reported next to the mean and the highest, each with its position and
    the number of tied positions (first position wins).
  • Historically conserved sequences follow the DiMA lineage (Hu et al. 2013,
    doi:10.1371/journal.pone.0059994; Abd Raman
    et al. 2020, doi:10.7717/peerj.7954): positions whose
    Index incidence reaches the threshold (above 50% and at most 100%, compared exactly) are
    joined when their k-mers overlap or abut and agree; regions that include positions with
    support ≤ T are flagged, and each region reports the exact 95% Clopper–Pearson lower bound
    of its conservation (Clopper & Pearson 1934,
    doi:10.1093/biomet/26.4.404).
    --hcs-eligible-only builds regions from positions with support above T only.
  • Alphabet: detected with Easel's rules as in HMMER (Eddy 2011,
    doi:10.1371/journal.pcbi.1002195) or set
    with --alphabet; lower case is folded to upper case; T and U are one symbol; gaps
    (- . _), missing data (~ ?), stops (* $ ^), frameshifts (#) and ambiguity codes
    (Cornish-Bowden 1985, doi:10.1093/nar/13.9.3021;
    Los Alamos HIV Sequence Database symbols) exclude the k-mers that contain them; protein U and
    O are residues.
  • Default k follows the alphabet: 9 for proteins, 27 for nucleotides. Any k from 1 to the
    alignment length is accepted.
  • Header metadata: values are split on |, cleaned (invisible characters removed, white
    space trimmed, Unicode NFC; Unicode Standard Annex #15) and empty values get the fill value.

Added

  • dima analyze, view, hcs, cite and completions subcommands; --run-report writes
    a separate JSON record of how a run went (versions, command, times, threads, resources,
    checksums) without identifying details.
  • Outputs: complete JSON with provenance, JSONL (one position per line), four tidy TSV tables
    (positions, variants, metadata, conserved sequences) and conserved-sequence JSON, each
    optionally compressed (.gz, .zst); JSON Schemas (Draft 2020-12) for every JSON output.
  • Typed diagnostics in the results and on standard error, each with a count, examples and a
    hint; --validation strict|permissive|report.
  • Input: plain, gzip/BGZF, bzip2 and zstd files (xz and other formats are refused with a hint
    on how to convert them), concatenated streams, standard input, pipes and FIFOs; the SHA-256
    of the stored input is recorded.
  • dima cite prints the references in text, BibTeX, RIS or CFF; --from results.json
    prints a methods paragraph and the references of that run. CITATION.cff and
    .zenodo.json describe the software.
  • Interruptions: SIGINT, SIGTERM and SIGHUP (and SIGUSR1, SIGUSR2, SIGXCPU and SIGALRM, which
    batch systems send before a limit) stop the run, remove unfinished outputs and end it by the
    same signal; outputs are written to temporary files and replace their destinations only when
    complete, keeping the permissions of a replaced file.
  • Batch-scheduler-aware default thread count (Slurm, PBS, SGE, LSF). When the system refuses
    some worker threads (a process, thread or container limit), the analysis continues with those
    it started and warns (threads_refused); the run report records both numbers.
  • Messages, --help (apart from authors' names) and the progress line are plain ASCII, so they
    read the same in any locale, pager, Windows console or screen reader;
    --progress auto|plain|off (or DIMA_PROGRESS) chooses between a line redrawn on terminals,
    appended lines for screen readers and logs, and nothing; colour follows --color,
    NO_COLOR, FORCE_COLOR and CLICOLOR_FORCE.
  • Numerical accuracy is measured and documented up to 17.5 million sequences per position:
    rounding error reaches at most 5e-11 relative, at least 10^10 times below the sampling error
    (Darriba et al. 2018,
    doi:10.1093/molbev/msy014). A fix for the last
    digits was prototyped and is deferred, because it would change stored digits without
    changing any scientific result.
  • Desktop application on the same library.

Changed

  • New command-line names (-s/--support-threshold, -a/--alphabet, --estimator,
    --query-name, --hcs-min-incidence, …); the support threshold defaults to 100 as in the
    paper.
  • Exit codes: 0 success, 1 internal error, 2 usage, 3 input/output or system error (such as a
    refused thread), 4 invalid input data, 130/143/129 interrupted.
  • Results are identical across operating systems, thread counts and batch sizes, and for
    wrapped, lower-case, CRLF or compressed copies of the same alignment.
  • Lines before the first record that are blank or hold only white space are skipped (DiMA
    v5.0.9 stopped at any line before the first header); other text before it, including an
    indented >, is an error with its line and column.
  • Memory grows with the diversity of the alignment, not with the number of sequences times
    its length.

Removed

  • The binary .dima format, memory-mapping and disk-backed matrix options
    (--low-memory, --force-ram, --temp-dir), --allow-lowercase, --report-invalid,
    --no-header and the record-length and sequence-count limits.

Fixed

  • Line-wrapped FASTA files were read with line breaks inside k-mers.
  • Lower-case residues were rejected, and malformed records were skipped silently.
  • Concatenated bzip2 streams were truncated after the first stream.
  • The equal-length check compared k-mer counts instead of alignment columns.
  • k was limited to 14 (protein) or 27 (nucleotide).
  • Several options had no effect; the desktop app and the command line disagreed on defaults
    and alphabets.

Install

Command-line tool (pick the archive for your system, unpack it and put dima on your PATH):

System Archive
Linux x86_64 (glibc 2.17 or newer: CentOS 7, Ubuntu, Debian, Rocky, most HPC clusters) dima-1.0.0-x86_64-unknown-linux-gnu.tar.gz
Linux ARM64 (glibc 2.28 or newer) dima-1.0.0-aarch64-unknown-linux-gnu.tar.gz
Linux, any distribution (static musl build; Alpine) dima-1.0.0-x86_64-unknown-linux-musl.tar.gz, dima-1.0.0-aarch64-unknown-linux-musl.tar.gz
macOS (Apple silicon / Intel) dima-1.0.0-aarch64-apple-darwin.tar.gz / dima-1.0.0-x86_64-apple-darwin.tar.gz
Windows (x86_64 / ARM64) dima-1.0.0-x86_64-pc-windows-msvc.zip / dima-1.0.0-aarch64-pc-windows-msvc.zip

The glibc builds are recommended on Linux; the musl builds need no system libraries.

tar xzf dima-1.0.0-x86_64-unknown-linux-gnu.tar.gz
./dima-1.0.0-x86_64-unknown-linux-gnu/dima --version

From source (Rust 1.85 or newer): cargo install --locked --git https://github.com/ShanWeera/dima-perf --tag v1.0.0 dima.

Desktop app: DiMA-1.0.0-universal.dmg (macOS), DiMA-1.0.0-x86_64.msi (Windows x86_64), and for Linux (glibc 2.28 or newer) DiMA-1.0.0-x86_64.AppImage, dima-gui_1.0.0_x86_64.deb or dima-gui-1.0.0-x86_64-linux.tar.gz, with the same names for ARM64 (aarch64 in place of x86_64). The packages are not code-signed; see the installation guide for first-run steps.

Verify a download: sha256sum -c SHA256SUMS --ignore-missing, and its build provenance: gh attestation verify <file> --repo ShanWeera/dima-perf. Each archive has a CycloneDX SBOM (.cdx.json), and the binaries embed their dependency list (cargo auditable).

Documentation for this version: dima-1.0.0-docs.tar.gz and https://shanweera.github.io/dima-perf/v1.0/

Cite: Tharanga S, et al. DiMA: sequence diversity dynamics analyser for viruses. Briefings in Bioinformatics. 2025;26(1):bbae607. doi:10.1093/bib/bbae607 (run dima cite for the software citation).

Allocator benchmark (musl + mimalloc vs glibc)

HIV1_FLT_2022_pol_DNA.fasta, 2637 sequences; runner: 4 CPUs. Medians of 3 runs.

Threads glibc wall (s) glibc peak RSS (MB) musl wall (s) musl peak RSS (MB)
1 1.91 105 2.21 113
8 1.07 121 1.36 203