Skip to content

v1.4.0 — comparesketch --low-mem streaming mode

Choose a tag to compare

@hamidamz hamidamz released this 16 Jul 18:50

Features

  • comparesketch --low-mem: streaming M×N comparison for large reference sets.
    The default mode pre-caches every reference's k-mer hashes (M+N file reads), but its
    memory scales with the entire reference universe — for a few thousand multi-megabase
    genomes this can exceed 1 TB. With --low-mem, only one Bloom filter and one
    reference's k-mers are held per thread at a time: each first-set reference is loaded
    into a filter, and every second-set reference is re-read from disk and queried against
    it. Memory becomes O(threads × genome) instead of O(all genomes), at the cost of
    M×N (rather than M+N) file reads. The similarity output is identical to the default
    mode; only the memory/I/O profile changes.

Full changelog: https://github.com/amazon-science/uniqsketch/blob/main/CHANGELOG.md