Repository navigation
First release of the Rust implementation of DiMA (Tharanga et al. 2025, Briefings in
Bioinformatics 26(1):bbae607, doi:10.1093/bib/bbae607).
It replaces every earlier version of this repository; results of those versions are not
comparable (see "Fixed").
Methodology
- Entropy correction (
--estimator dima, the default) follows the paper's Fig. 2 exactly:
subsample sizes of R% of the sequences, with R from max(50, ⌈100·T/N⌉) rounded up to a
multiple of 5 in steps of 5, plus 100%; sizes ⌈R·N/100⌉; sampling with replacement; the
entropy regressed on 1/n; the intercept is the estimate, with the paper's fallbacks (raw
entropy when N ≤ T or with fewer than 5 points). It is computed as the
exact expectation of the random intercept, so results are reproducible bit for bit, with the
random intercept's mean and no resampling noise (Rao 1945; Blackwell 1947,
doi:10.1214/aoms/1177730497): the limit of
averaging many random subsets, as DiMA's predecessors did (Khan et al. 2017,
doi:10.1186/s12920-017-0301-2). The expected
intercept is never negative, so the paper's negative-intercept rule never applies; DiMA
v5.0.9 applied it to each random run, which raised its average at near-conserved positions
(MERS-CoV spike protein eligible mean 0.126 bits here, about 0.142 averaged over v5.0.9 runs;
HIV-1 pol unchanged). Every reported entropy is checked to be a finite number of at least 0
bits; a violation is an internal error (exit 1), never replaced by another value. On the paper's HIV-1 pol dataset the eligible mean is 5.3504 bits (paper: ~5.35). - Alternative estimators:
chao-wang-jost(Chao, Wang & Jost 2013,
doi:10.1111/2041-210X.12108),miller-madow
(Miller 1955) andraw(plug-in). Each position records the estimator or fallback used.
The second term of Chao–Wang–Jost is evaluated exactly in a time independent of the number of
sequences (hypergeometric series or Gauss's continued fraction: Hankin 2015,
doi:10.32614/RJ-2015-022; Van Vleck 1901,
doi:10.2307/1967627; Henrici & Pflüger 1966,
doi:10.1007/BF02166031; Jones & Thron 1974,
doi:10.1090/S0025-5718-1974-0373265-5;
Alzer 1997, doi:10.1090/S0025-5718-97-00807-7),
where summing it over every sequence would cost about 10% of a run and lose up to 8 digits. - Sample coverage per position (Chao & Jost 2012, Eq. 4a,
doi:10.1890/11-1952.1), with an optional
--min-coverageflag. - Positions without support (NS) have no entropy (
null/NA), never 0 bits; summaries
cover positions with support above T only, with the paper's averaging also reported as
all_positions_summary. - Lowest entropy is reported next to the mean and the highest, each with its position and
the number of tied positions (first position wins). - Historically conserved sequences follow the DiMA lineage (Hu et al. 2013,
doi:10.1371/journal.pone.0059994; Abd Raman
et al. 2020, doi:10.7717/peerj.7954): positions whose
Index incidence reaches the threshold (above 50% and at most 100%, compared exactly) are
joined when their k-mers overlap or abut and agree; regions that include positions with
support ≤ T are flagged, and each region reports the exact 95% Clopper–Pearson lower bound
of its conservation (Clopper & Pearson 1934,
doi:10.1093/biomet/26.4.404).
--hcs-eligible-onlybuilds regions from positions with support above T only. - Alphabet: detected with Easel's rules as in HMMER (Eddy 2011,
doi:10.1371/journal.pcbi.1002195) or set
with--alphabet; lower case is folded to upper case; T and U are one symbol; gaps
(- . _), missing data (~ ?), stops (* $ ^), frameshifts (#) and ambiguity codes
(Cornish-Bowden 1985, doi:10.1093/nar/13.9.3021;
Los Alamos HIV Sequence Database symbols) exclude the k-mers that contain them; protein U and
O are residues. - Default k follows the alphabet: 9 for proteins, 27 for nucleotides. Any k from 1 to the
alignment length is accepted. - Header metadata: values are split on
|, cleaned (invisible characters removed, white
space trimmed, Unicode NFC; Unicode Standard Annex #15) and empty values get the fill value.
Added
dima analyze,view,hcs,citeandcompletionssubcommands;--run-reportwrites
a separate JSON record of how a run went (versions, command, times, threads, resources,
checksums) without identifying details.- Outputs: complete JSON with provenance, JSONL (one position per line), four tidy TSV tables
(positions, variants, metadata, conserved sequences) and conserved-sequence JSON, each
optionally compressed (.gz,.zst); JSON Schemas (Draft 2020-12) for every JSON output. - Typed diagnostics in the results and on standard error, each with a count, examples and a
hint;--validation strict|permissive|report. - Input: plain, gzip/BGZF, bzip2 and zstd files (xz and other formats are refused with a hint
on how to convert them), concatenated streams, standard input, pipes and FIFOs; the SHA-256
of the stored input is recorded. dima citeprints the references in text, BibTeX, RIS or CFF;--from results.json
prints a methods paragraph and the references of that run.CITATION.cffand
.zenodo.jsondescribe the software.- Interruptions: SIGINT, SIGTERM and SIGHUP (and SIGUSR1, SIGUSR2, SIGXCPU and SIGALRM, which
batch systems send before a limit) stop the run, remove unfinished outputs and end it by the
same signal; outputs are written to temporary files and replace their destinations only when
complete, keeping the permissions of a replaced file. - Batch-scheduler-aware default thread count (Slurm, PBS, SGE, LSF). When the system refuses
some worker threads (a process, thread or container limit), the analysis continues with those
it started and warns (threads_refused); the run report records both numbers. - Messages,
--help(apart from authors' names) and the progress line are plain ASCII, so they
read the same in any locale, pager, Windows console or screen reader;
--progress auto|plain|off(orDIMA_PROGRESS) chooses between a line redrawn on terminals,
appended lines for screen readers and logs, and nothing; colour follows--color,
NO_COLOR,FORCE_COLORandCLICOLOR_FORCE. - Numerical accuracy is measured and documented up to 17.5 million sequences per position:
rounding error reaches at most 5e-11 relative, at least 10^10 times below the sampling error
(Darriba et al. 2018,
doi:10.1093/molbev/msy014). A fix for the last
digits was prototyped and is deferred, because it would change stored digits without
changing any scientific result. - Desktop application on the same library.
Changed
- New command-line names (
-s/--support-threshold,-a/--alphabet,--estimator,
--query-name,--hcs-min-incidence, …); the support threshold defaults to 100 as in the
paper. - Exit codes: 0 success, 1 internal error, 2 usage, 3 input/output or system error (such as a
refused thread), 4 invalid input data, 130/143/129 interrupted. - Results are identical across operating systems, thread counts and batch sizes, and for
wrapped, lower-case, CRLF or compressed copies of the same alignment. - Lines before the first record that are blank or hold only white space are skipped (DiMA
v5.0.9 stopped at any line before the first header); other text before it, including an
indented>, is an error with its line and column. - Memory grows with the diversity of the alignment, not with the number of sequences times
its length.
Removed
- The binary
.dimaformat, memory-mapping and disk-backed matrix options
(--low-memory,--force-ram,--temp-dir),--allow-lowercase,--report-invalid,
--no-headerand the record-length and sequence-count limits.
Fixed
- Line-wrapped FASTA files were read with line breaks inside k-mers.
- Lower-case residues were rejected, and malformed records were skipped silently.
- Concatenated bzip2 streams were truncated after the first stream.
- The equal-length check compared k-mer counts instead of alignment columns.
- k was limited to 14 (protein) or 27 (nucleotide).
- Several options had no effect; the desktop app and the command line disagreed on defaults
and alphabets.
Install
Command-line tool (pick the archive for your system, unpack it and put dima on your PATH):
| System | Archive |
|---|---|
| Linux x86_64 (glibc 2.17 or newer: CentOS 7, Ubuntu, Debian, Rocky, most HPC clusters) | dima-1.0.0-x86_64-unknown-linux-gnu.tar.gz |
| Linux ARM64 (glibc 2.28 or newer) | dima-1.0.0-aarch64-unknown-linux-gnu.tar.gz |
| Linux, any distribution (static musl build; Alpine) | dima-1.0.0-x86_64-unknown-linux-musl.tar.gz, dima-1.0.0-aarch64-unknown-linux-musl.tar.gz |
| macOS (Apple silicon / Intel) | dima-1.0.0-aarch64-apple-darwin.tar.gz / dima-1.0.0-x86_64-apple-darwin.tar.gz |
| Windows (x86_64 / ARM64) | dima-1.0.0-x86_64-pc-windows-msvc.zip / dima-1.0.0-aarch64-pc-windows-msvc.zip |
The glibc builds are recommended on Linux; the musl builds need no system libraries.
tar xzf dima-1.0.0-x86_64-unknown-linux-gnu.tar.gz
./dima-1.0.0-x86_64-unknown-linux-gnu/dima --versionFrom source (Rust 1.85 or newer): cargo install --locked --git https://github.com/ShanWeera/dima-perf --tag v1.0.0 dima.
Desktop app: DiMA-1.0.0-universal.dmg (macOS), DiMA-1.0.0-x86_64.msi (Windows x86_64), and for Linux (glibc 2.28 or newer) DiMA-1.0.0-x86_64.AppImage, dima-gui_1.0.0_x86_64.deb or dima-gui-1.0.0-x86_64-linux.tar.gz, with the same names for ARM64 (aarch64 in place of x86_64). The packages are not code-signed; see the installation guide for first-run steps.
Verify a download: sha256sum -c SHA256SUMS --ignore-missing, and its build provenance: gh attestation verify <file> --repo ShanWeera/dima-perf. Each archive has a CycloneDX SBOM (.cdx.json), and the binaries embed their dependency list (cargo auditable).
Documentation for this version: dima-1.0.0-docs.tar.gz and https://shanweera.github.io/dima-perf/v1.0/
Cite: Tharanga S, et al. DiMA: sequence diversity dynamics analyser for viruses. Briefings in Bioinformatics. 2025;26(1):bbae607. doi:10.1093/bib/bbae607 (run dima cite for the software citation).
Allocator benchmark (musl + mimalloc vs glibc)
HIV1_FLT_2022_pol_DNA.fasta, 2637 sequences; runner: 4 CPUs. Medians of 3 runs.
| Threads | glibc wall (s) | glibc peak RSS (MB) | musl wall (s) | musl peak RSS (MB) |
|---|---|---|---|---|
| 1 | 1.91 | 105 | 2.21 | 113 |
| 8 | 1.07 | 121 | 1.36 | 203 |