BitMagic 9.3.1

BitMagic 9.3.1

BitMagic 9.3.1 expands support for memory-efficient storage and selective retrieval of compressed data. This release introduces direct file streaming, persistent deserialization indexes, faster sparse-vector traversal, and shared NULL planes for related columns.

New functionality

• Direct streaming serialization to disk. Write containers to seekable binary C++ streams without first constructing a complete serialized BLOB in RAM. Supported containers include bm::bvector<>, bm::sparse_vector<>, bm::rsc_sparse_vector<>, bm::str_sparse_vector<>, and bm::sparse_vector_float<>. Reusable working buffers reduce temporary memory requirements under memory pressure. Stream output is byte-for-byte equivalent to RAM serialization with the same settings.
• Stream-based deserialization. Restore containers directly from seekable binary input streams without loading the entire serialized BLOB into a temporary RAM buffer. Stream interfaces also support selective restoration and deserialization-index construction.
• Index-assisted gather deserialization. Retrieve selected logical positions from serialized bit-vectors and sparse vectors. A selection bit-vector identifies requested positions, while bookmarks and a deserialization index help the decoder locate relevant blocks and skip unrelated compressed payloads.
• Selective retrieval from memory-mapped files. Gather deserialization supports memory-mapped BLOBs, allowing applications to keep large models and data-frame columns compressed on SSD and reconstruct selected subsets on demand. Sparse retrieval can reduce resident memory, decoding work, and accesses to unrelated mapped pages. Access is block-oriented, and requested values are decoded into a destination container.
• Persistent deserialization indexes. Build a navigation index once, save it, and restore it for subsequent retrievals. The index records serialized marker offsets and bookmark landing points; sparse-vector indexes organize this information across their bit-planes. Persistence avoids rescanning the source BLOB to rebuild the index on later runs. Versioned, compressed index serialization supports bit-vectors, sparse vectors, and floating-point sparse vectors. An index describes a particular serialized BLOB and must be rebuilt when that representation changes.
• New bm::for_each_sparse() algorithms. Traverse integer, rank-select compressed, and floating-point sparse vectors using reusable decoding buffers. Full traversal decodes values in chunks; a filtered overload uses a selection bit-vector and batched gather() calls. Callbacks receive (value, is_null, index), with support for configurable buffer sizes and early termination.
• Shared NULL planes. Related sparse-vector columns can share a common NOT NULL bit-vector, reducing duplicated validity information. Serialization can omit a follower's externally owned NULL plane. Connected columns are restored by loading the owner first and attaching followers before deserialization. RSC followers can also reuse the owner's rank-select index.
• Explicit logical-size preservation. Sparse-vector serialization now preserves the complete logical extent, including trailing NULL positions in rank-select compressed vectors.

Performance improvements and bug fixes

• Improved bookmark positioning and block skipping for repeated sparse gathers, including access to memory-mapped data.
• Improved rank decompression to use an available rank-select index.
• Optimized floating-point iteration and chunked traversal, including shared rank/count work when decoding RSC-backed exponent and mantissa components.
• Fixed sparse-vector deserialization to constrain restored value and NULL planes to the appropriate logical extent or requested range.
• Fixed handling of fully populated bit-vector superblocks in block-management code used by deserialization.
• Corrected construction and deserialization of floating-point sparse vectors with shared exponent/mantissa NULL information, including RSC-backed vectors.
• Improved GCC/Linux build compatibility and resolved compiler warnings.
• Updated native macOS/Xcode build configuration.
• Consolidated parallel stress-test runners and expanded regression and performance coverage for streaming I/O, indexed gathers, persistent indexes, shared NULL planes, and memory-mapped retrieval.

Headers to explore

Header	Functionality
bmfio.h — new	C++ stream I/O: streams_encoder, streams_decoder, streams_deserializer<>, streams_sparse_vector_deserializer<>, and streams_sparse_vector_float_deserializer<>.
bmserial.h	Bit-vector streaming support, indexed selective deserialization, and deserialization_index<> persistence.
bmsparsevec_serial.h	Sparse-vector gather deserialization, persistent indexes, and logical-size serialization.
bmsparsevec_float_serial.h	Floating-point sparse-vector serialization, gather deserialization, and persistent indexes.
bmsparsevec_algo.h	Full and filtered bm::for_each_sparse() traversal.

New examples

• bvsample27 — Bit-vector file I/O and indexed gather. Streaming serialization/deserialization, serializer reuse, saving and restoring deserialization indexes, and selective retrieval from files and RAM BLOBs.
• rscsample07 — Shared NULL planes. Connected RSC and plain sparse-vector columns, shared validity information, serialization, and consistent range restoration.
• strsvsample10 — Memory-mapped string-vector retrieval. Streaming a string vector to disk, memory mapping the serialized file, and repeated index-assisted gathers.
• svfsample05 — Search results and deferred float retrieval. Saving a floating-point vector, selection bit-vectors, and a deserialization index, then restoring selected values and computing aggregate statistics.

Expanded examples and documentation

• svsample04 and rscsample03: bm::for_each_sparse() traversal.
• svfsample02: floating-point traversal and streaming file serialization/deserialization.
• svsample08: index-assisted selective sparse-vector restoration.
• Added individual READMEs throughout the non-xsample collection and expanded the sample catalogue with cross-references.

Compatibility notes

• New sparse-vector writers emit schema version 3 for 32-bit addressing or 4 for BM64ADDR, carrying explicit logical-size information. New readers retain support for older BLOBs; older readers are not guaranteed to accept the new format.
• The C++ file-stream adapters require seekable binary streams. Output streams must not use append mode because serialization patches headers and bookmarks.

Full changelog: v9.2.1...v9.3.1