kbf_compression

Compression of bloom filters from kmindex using parallel processing with background rsync.

Requirements

• Python 3.6+
• main_bitmatrixshuffle binary in PATH (or specify with --bitmatrix-shuffle)
• rsync for background file synchronization

Usage

python compress_watcher.py <permutation_file> <index_id> [options]

Arguments

Argument	Description
permutation_file	Path to permutation binary file
index_id	Index ID (e.g., species_name)

Options

Option	Default	Description
--matrix-base	./index_data	Path to matrix base directory
--output-folder	./compressed	Path to output folder
--bitmatrix-shuffle	main_bitmatrixshuffle	Path to bitmatrix shuffle binary
--max-jobs	16	Maximum number of parallel compression jobs
--num-matrices	256	Total number of matrices to compress

Example

python compress_watcher.py /path/to/permutation.bin my_index \
    --matrix-base /data/kmindex \
    --output-folder /data/compressed \
    --max-jobs 8

How it works

1. Starts a background rsync to copy matrix files from source to output directory
2. Monitors for matrix files reaching 75% of expected size
3. Submits compression jobs in parallel (up to --max-jobs concurrent)
4. Uses main_bitmatrixshuffle to compress each matrix file
5. Outputs compressed files as blocks{N} in the output matrices directory