Repository navigation
Version 2.0.0 built on 2026-11-18
This major release introduces some small Python wrapper API breaks, comes with major 2-8x performance improvements over 1.7.0 depending on dataset and thread count, and is the first release using AI assistance.
The plots contain indexed_bzip2/ibzip2 1.7.0/2.0.0, GNU bzip2, many other old and new bzip2 decompressors, and even GNU gzip and igzip for comparison, as measured on a recompressed gzip version of the dataset. On data with high compression ratios, indexed_bzip2 is much faster than GNU gzip. For low compression ratios, such as FASTQ data or Silesia, it is roughly at GNU gzip parity and much slower than igzip.
Here you can find the full benchmarks
Here you can find a kind of blog post about this release
Features
- Add parallel bzip2 compression to the
ibzip2CLI. - Add
--count,--count-lines, byte--ranges, and--import-index/--export-indexto theibzip2CLI. - Stream the bzip2 block offset index through
-L/--export-indexwhile decoding and compressing.--list-offsetsremains an alias for compatibility. The compression-side scan works from the in-memory chunk, so it also supports writing compressed data to stdout. Empty streams get an EOS-plus-sentinel index, and chunks whose inner block offsets cannot be derived are rejected instead of writing an incomplete index. - Add
import_indexandexport_indextoIndexedBzip2Filefor the plain text bzip2 block offset index.
Performance
- Reuse the bzip2 block decoder state and
BlockDatabuffers across blocks (1.01x single-core, 1.03x text64 at P4). - Decode run records and invert the BWT with interleaved chains instead of libsais (1.32x Silesia single-core, Huffman 0.74 s -> 0.64 s, and inverse BWT 0.64 s -> 0.41 s).
- Replace the eight-pass byte-shifted
std::string_view::findblock-magic search with a single word/SIMD scan and runtime dispatch to portable, SSE2, AVX2, or AVX-512 kernels (about 2x finder bandwidth on cache-resident data and 1.5x on 64 MiB). - Add SSE4.1, AVX2, and AVX-512 symbol decoding kernels, selected per block from the Huffman code lengths (1.23x Silesia and 1.26x text64 single-core).
- Dispatch the last-to-first mapping builder to AVX2 or AVX-512 at runtime so prebuilt wheels use the wider stores on capable x86 CPUs.
- Pack the Huffman LUT entry and MTF list (1.07x text decode).
- Make the MTF update branchless (1.28x base64 decode).
- Compute the bzip2 CRC with slicing-by-N tables, and with PCLMUL/VPCLMULQDQ where available (4.2x on constant data, 2.9-3.0x on RLE boundary data, 1.9x on high-ratio files).
- Allocate parallel bzip2 blocks at their exact decoded size and prefetch the last block (3.1x on constant data and 4.0x on RLE boundary data with eight physical cores).
- Decode the next Huffman symbol ahead of the MTF update (1.05-1.10x on text, DNA, and random data).
- Use NEON for the bzip2 MTF and RLE expansion, symbol kernel, and last-to-first mapping builder on ARM.
- Decode the sequential reader in 64 KiB chunks, emit RLE1 runs incrementally, store short runs with one word write, and add a 64-byte AVX-512 RLE scan.
- Keep the block decoder state off the worker stack, avoid zero-initializing and reuse the per-block BWT buffers, and parse the next block header in the worker.
- Avoid
RawIOBase.readall's 8 KiB chunks and reallocations in the Python wrapper. - Fix a performance regression when repeatedly reading less than the chunk size per call.
- Use the physical core count for automatic
ibzip2parallelization because it scales not well beyond.
Fixes
- Return
NonefromIndexedBzip2File.size()when the uncompressed size is not known yet instead of returning 0. - Make
ibzip2 --testwork without also passing--decompress. - Honor the requested block-finder parallelism in
ibzip2, and keep explicit output when--testis combined with an output path. - Enable 64-bit file offsets on 32-bit builds so files larger than 2/4 GiB can be read and indexed.
- Reject bzip2 run lengths that overflow the block size and data blocks with an empty symbol map.
- Distinguish empty from truncated bzip2 streams and verify the bzip2 stream CRC in the parallel reader. Accept EOF immediately after a complete EOS on non-seekable inputs instead of reporting the next stream as truncated.
- Report a zero compressed offset from
ParallelBZ2Reader::tellCompressedbefore the first read. - Decode concatenated bzip2 streams and verify the CRCs of imported indexes while decoding.
- Reject imported bzip2 block indexes whose last block offsets are incomplete, whose decoded block sizes disagree with the data, or that cover only the first stream of a concatenated file.
- Share the file-reader and core fixes from the rapidgzip core, including
BufferedFileReaderseek and cache behavior, non-seekable stream sizes, null-buffer skips, out-of-rangeBufferViewFileReaderseeks, and resetting the eviction strategy inclear(). - Fix reads from Python file-like objects returning more data than requested and require writable C-contiguous buffers for Python
readinto. - Support non-seekable Python file objects without
fileno, keep caller-owned file descriptors open while detecting the input type, release callback references, propagate Python exceptions from file-like objects, make reader construction and close exception-safe, and release the GIL while joining the block finder. - Accept path-like and bytes objects for the file name in the standalone and parallel readers.
- Fix writing to
/dev/nullwhen the file size cannot be determined. - Treat
-as stdin or stdout in theibzip2CLI, and avoid decoding the whole input just to print verbose range information. - Retry interrupted writes and avoid a lost reader-thread wakeup for non-seekable inputs. Check the compressor chunk length before scanning for a block magic and restrict the packed-MTF NEON path to little-endian targets.
- Validate parallel bzip2 block offsets before updating the block finder, reject invalid Huffman codes and block selectors with descriptive errors, and do not require the decoded block to end before the next candidate.
- Fix block-zero prefetching, keep rethrown worker exceptions alive in the shared block fetcher, and
keep the parallel readers usable after joining threads. - Report a non-seekable empty input as EOF, reject imported indexes that omit the first data block, and verify the stream CRC after seeking back to the start.
- Propagate buffer-acquisition failures and conversion errors from Python file-like objects, and
validate the requested parallelization before starting worker threads.
API
- Widen file, bit, block, and chunk offsets to 64 bit where they were still
size_t. Block finders now returnstd::optional<uint64_t>instead of usingstd::numeric_limits::max()sentinels, and the compression block-offset callback takes 64-bit offsets. - Add type annotations to the Python wrappers using the
FileOrPath,IndexFile, andWritableBufferaliases. - Remove the
filenomethod from the Python readers.
Binaries
The offered binaries are statically built and therefore should run on most systems released in the last 10 years without additional dependencies.
The package binaries can simply be downloaded, extracted, and executed.
The binaries contain a command-line interface similar to bzip2. See, e.g., ibzip2 --help
Alternatively, installation helpers such as eget and ubi can be used.
eget --to="$HOME/.local/bin/" mxmlnkn/librapidarchiveoreget --pre-release mxmlnkn/librapidarchiveubi -i "$HOME/.local/bin/" -p mxmlnkn/librapidarchive(seems to not work with pre-releases)


