Skip to content

indexed_bzip2 2.0.0

Latest

Choose a tag to compare

@mxmlnkn mxmlnkn released this 18 Nov 01:07
· 5 commits to master since this release

Version 2.0.0 built on 2026-11-18

This major release introduces some small Python wrapper API breaks, comes with major 2-8x performance improvements over 1.7.0 depending on dataset and thread count, and is the first release using AI assistance.

The plots contain indexed_bzip2/ibzip2 1.7.0/2.0.0, GNU bzip2, many other old and new bzip2 decompressors, and even GNU gzip and igzip for comparison, as measured on a recompressed gzip version of the dataset. On data with high compression ratios, indexed_bzip2 is much faster than GNU gzip. For low compression ratios, such as FASTQ data or Silesia, it is roughly at GNU gzip parity and much slower than igzip.

Bzip2 decompression bandwidth comparison on Ryzen 9 3900X on CTU-13 dataset

Bzip2 decompression bandwidth comparison on EPYC 7702 on CTU-13 dataset

Bzip2 decompression bandwidth comparison on Ryzen 9 3900X on Silesia dataset

Here you can find the full benchmarks

Here you can find a kind of blog post about this release

Features

  • Add parallel bzip2 compression to the ibzip2 CLI.
  • Add --count, --count-lines, byte --ranges, and --import-index/--export-index to the ibzip2 CLI.
  • Stream the bzip2 block offset index through -L/--export-index while decoding and compressing. --list-offsets remains an alias for compatibility. The compression-side scan works from the in-memory chunk, so it also supports writing compressed data to stdout. Empty streams get an EOS-plus-sentinel index, and chunks whose inner block offsets cannot be derived are rejected instead of writing an incomplete index.
  • Add import_index and export_index to IndexedBzip2File for the plain text bzip2 block offset index.

Performance

  • Reuse the bzip2 block decoder state and BlockData buffers across blocks (1.01x single-core, 1.03x text64 at P4).
  • Decode run records and invert the BWT with interleaved chains instead of libsais (1.32x Silesia single-core, Huffman 0.74 s -> 0.64 s, and inverse BWT 0.64 s -> 0.41 s).
  • Replace the eight-pass byte-shifted std::string_view::find block-magic search with a single word/SIMD scan and runtime dispatch to portable, SSE2, AVX2, or AVX-512 kernels (about 2x finder bandwidth on cache-resident data and 1.5x on 64 MiB).
  • Add SSE4.1, AVX2, and AVX-512 symbol decoding kernels, selected per block from the Huffman code lengths (1.23x Silesia and 1.26x text64 single-core).
  • Dispatch the last-to-first mapping builder to AVX2 or AVX-512 at runtime so prebuilt wheels use the wider stores on capable x86 CPUs.
  • Pack the Huffman LUT entry and MTF list (1.07x text decode).
  • Make the MTF update branchless (1.28x base64 decode).
  • Compute the bzip2 CRC with slicing-by-N tables, and with PCLMUL/VPCLMULQDQ where available (4.2x on constant data, 2.9-3.0x on RLE boundary data, 1.9x on high-ratio files).
  • Allocate parallel bzip2 blocks at their exact decoded size and prefetch the last block (3.1x on constant data and 4.0x on RLE boundary data with eight physical cores).
  • Decode the next Huffman symbol ahead of the MTF update (1.05-1.10x on text, DNA, and random data).
  • Use NEON for the bzip2 MTF and RLE expansion, symbol kernel, and last-to-first mapping builder on ARM.
  • Decode the sequential reader in 64 KiB chunks, emit RLE1 runs incrementally, store short runs with one word write, and add a 64-byte AVX-512 RLE scan.
  • Keep the block decoder state off the worker stack, avoid zero-initializing and reuse the per-block BWT buffers, and parse the next block header in the worker.
  • Avoid RawIOBase.readall's 8 KiB chunks and reallocations in the Python wrapper.
  • Fix a performance regression when repeatedly reading less than the chunk size per call.
  • Use the physical core count for automatic ibzip2 parallelization because it scales not well beyond.

Fixes

  • Return None from IndexedBzip2File.size() when the uncompressed size is not known yet instead of returning 0.
  • Make ibzip2 --test work without also passing --decompress.
  • Honor the requested block-finder parallelism in ibzip2, and keep explicit output when --test is combined with an output path.
  • Enable 64-bit file offsets on 32-bit builds so files larger than 2/4 GiB can be read and indexed.
  • Reject bzip2 run lengths that overflow the block size and data blocks with an empty symbol map.
  • Distinguish empty from truncated bzip2 streams and verify the bzip2 stream CRC in the parallel reader. Accept EOF immediately after a complete EOS on non-seekable inputs instead of reporting the next stream as truncated.
  • Report a zero compressed offset from ParallelBZ2Reader::tellCompressed before the first read.
  • Decode concatenated bzip2 streams and verify the CRCs of imported indexes while decoding.
  • Reject imported bzip2 block indexes whose last block offsets are incomplete, whose decoded block sizes disagree with the data, or that cover only the first stream of a concatenated file.
  • Share the file-reader and core fixes from the rapidgzip core, including BufferedFileReader seek and cache behavior, non-seekable stream sizes, null-buffer skips, out-of-range BufferViewFileReader seeks, and resetting the eviction strategy in clear().
  • Fix reads from Python file-like objects returning more data than requested and require writable C-contiguous buffers for Python readinto.
  • Support non-seekable Python file objects without fileno, keep caller-owned file descriptors open while detecting the input type, release callback references, propagate Python exceptions from file-like objects, make reader construction and close exception-safe, and release the GIL while joining the block finder.
  • Accept path-like and bytes objects for the file name in the standalone and parallel readers.
  • Fix writing to /dev/null when the file size cannot be determined.
  • Treat - as stdin or stdout in the ibzip2 CLI, and avoid decoding the whole input just to print verbose range information.
  • Retry interrupted writes and avoid a lost reader-thread wakeup for non-seekable inputs. Check the compressor chunk length before scanning for a block magic and restrict the packed-MTF NEON path to little-endian targets.
  • Validate parallel bzip2 block offsets before updating the block finder, reject invalid Huffman codes and block selectors with descriptive errors, and do not require the decoded block to end before the next candidate.
  • Fix block-zero prefetching, keep rethrown worker exceptions alive in the shared block fetcher, and
    keep the parallel readers usable after joining threads.
  • Report a non-seekable empty input as EOF, reject imported indexes that omit the first data block, and verify the stream CRC after seeking back to the start.
  • Propagate buffer-acquisition failures and conversion errors from Python file-like objects, and
    validate the requested parallelization before starting worker threads.

API

  • Widen file, bit, block, and chunk offsets to 64 bit where they were still size_t. Block finders now return std::optional<uint64_t> instead of using std::numeric_limits::max() sentinels, and the compression block-offset callback takes 64-bit offsets.
  • Add type annotations to the Python wrappers using the FileOrPath, IndexFile, and WritableBuffer aliases.
  • Remove the fileno method from the Python readers.

Binaries

The offered binaries are statically built and therefore should run on most systems released in the last 10 years without additional dependencies.
The package binaries can simply be downloaded, extracted, and executed.
The binaries contain a command-line interface similar to bzip2. See, e.g., ibzip2 --help

Alternatively, installation helpers such as eget and ubi can be used.

  • eget --to="$HOME/.local/bin/" mxmlnkn/librapidarchive or eget --pre-release mxmlnkn/librapidarchive
  • ubi -i "$HOME/.local/bin/" -p mxmlnkn/librapidarchive (seems to not work with pre-releases)