Add sparse & compressed search index blocks

[klauspost]

A stream can contain search indexes only. This means that blocks are referenced into another stream.

Adds chunk type 0x46: Compressed Block Search Table

Allows storing search indexes with entropy compression.

This will allow higher precision tables to be stored without loosing precision - other than the size increase from fewer collisions.

We allow for flexibility in the encoder while always keeping decoding fast.

huff0 is about 500MB/s encoding and 2000MB/s decoding per thread and doesn't rely on any special CPU to be reasonably fast.

Adds chunk type "0x47: Remote Block Reference" to spec.

λ go build&&mz c -search -search.compress -search.lim=5 cockroach.node1.log && mz search -v "abcderterteartertfdgert" cockroach.node1.log.mz
Compressing cockroach.node1.log -> cockroach.node1.log.mz 10506623721 -> 999484530 [9.51% 10.512:1]; 3840.4MB/s
cockroach.node1.log.mz took 645ms 16289.3 MB/s
Blocks total: 1253, skipped: 1253 (100.0%), deferred: 17 (1.4%, 17 skipped)
Blocks searched: 0 (0.0%), false positive: 0 (0.0%)
Bytes skipped: 706028650 compressed, searched: 0 uncompressed
Tables: 1253 present, 0 missing, 0 unusable
Table bits/byte: 0.2234, log2: 23.0, avg reductions: 0.0
Table total: 293447084 bytes, avg 234195 bytes/table, 2.79% of 10506623721 uncompressed
Table population: avg 2.9%, min 1.9%, max 4.8%
Table types: 0 uncompressed (0x45), 1253 compressed (0x46)
Compressed tables: 1253 (100.0% of total), 293447084 wire bytes, 1313865728 uncompressed bitmap bytes (22.33% ratio)
huff0 sub-blocks: 20048 total (19916 tabled, 0 raw, 0 RLE, 132 sparse); 14129 tables emitted (share=0.71 tables/tabled-block)
  payload bytes: tabled=291446563 raw=0 RLE=0 sparse=1353246; table-header bytes=609685

λ go build&&mz c -search -search.compress -search.lim=5 -search.prefix="a" "cockroach.node1.log"&&mz search -v "abcderterteartertfdgert" "cockroach.node1.log.mz"
Compressing cockroach.node1.log -> cockroach.node1.log.mz 10506623721 -> 707476550 [6.73% 14.851:1]; 4283.9MB/s
cockroach.node1.log.mz took 74ms 141981.4 MB/s
Blocks total: 1253, skipped: 1252 (99.9%), deferred: 2 (0.2%, 2 skipped)
Blocks searched: 1 (0.1%), false positive: 1 (100.0%)
Bytes skipped: 705393872 compressed, searched: 8388608 uncompressed
Tables: 1253 present, 0 missing, 0 unusable
Table bits/byte: 0.0011, log2: 16.9, avg reductions: 6.1
Table total: 1439128 bytes, avg 1148 bytes/table, 0.01% of 10506623721 uncompressed
Table population: avg 0.7%, min 0.5%, max 1.0%
Table types: 0 uncompressed (0x45), 1253 compressed (0x46)
Compressed tables: 1253 (100.0% of total), 1439128 wire bytes, 20627456 uncompressed bitmap bytes (6.98% ratio)
huff0 sub-blocks: 1254 total (0 tabled, 0 raw, 0 RLE, 1254 sparse); 0 tables emitted (share=0.00 tables/tabled-block)
  payload bytes: tabled=0 raw=0 RLE=0 sparse=1410308; table-header bytes=0

New Blocks

2.2 Compressed Block Search Table (chunk type 0x46, skippable)

The Compressed Block Search Table is an optional chunk that will come before a block that represents the contents.

Length	Code	Description
1		Table Type
1		Search pattern length
1		Base Table Size in log2
0/8/32/1+n		Prefix values
1		Reductions from Base Table
4		CRC32 of table entries
1	h0_bs	log2 huff0 block size
1	h0_tc	huff0 table count
...		huff0 tables
1	h0_ti	huff0 table index
1-3	bd	compressed length (uvarint)
bd		huff0 4X compressed block data
...		Additional huff0 blocks

See Section 2.1 on how to decode table information until the huff0 section.

The index bits are split into huff0 blocks.

The first value (h0_bs) is the log2 of the uncompressed huff0 block size in bytes.
The bitmap size (n) must be divisible by the huff0 block size.
The maximum huff0 block size is 128KiB (h0_bs = 17).
The minimum size is 32 bytes (h0_bs = 5).

Tables for huff0 blocks are stored separately. There can be up to 16 tables per block.
The encoder is allowed to reuse tables for different huff0 blocks.

The encoded huff0 tables follow. This is similar to zstd, rfc 8878.
Table sizes are self-contained, but h0_tc must be read.

The number of blocks can be calculated as n / (1 << h0_bs).

For each block the table index is specified by the h0_ti value and decoded as follows:

h0_ti value	Meaning
0 -> 15	huff0 table index
16	Uncompressed block
17	RLE - Read 1 byte, repeat value for block
18	Sparse Bit Table
19 -> 255	Reserved [invalid]

If h0_ti is <= 15, the compressed size follows as a uvarint - and the compressed data itself.
Note the compressed streams are always 4 streams interleaved,
also named 4X in zstd.

An uncompressed block (h0_ti = 16) and RLE (h0_ti = 17) will not have any size,
since that can be inferred.

RLE in this context means "single value repeated for the entire block" and can only be used for that.

2.2.1 Sparse Bit Table

A sparse bit table can be used for very sparsely populated blocks.

Like huff0 block, this starts with a uvarint encoded length in bytes.

The block contains byte-encoded distances between bits.

To decode, read single bytes. Each byte is the distance to the next set bit.
Bits are counted from the least significant bit.

If the byte is 255, add 255 to the distance and read the following byte.

When the final byte has been read, there are no more bits.
The distance is not expected to reach the end of the block.

2.3 Remote Block Reference (chunk type 0x47, skippable)

When generating search indexes from an existing stream or you, for
another reason, want to separate the search index from the data,
you can use the Remote Block Reference chunk type to indicate a remote block.

Length	Description
UVarInt	Block Offset
...	[Additional relative block offsets]

This block replaces a block that and indicates the offset of the block in the stream.

If more values are present, it indicates additional blocks without indexes between.
These are stored as relative offsets from the current block.

The block offsets must be in strictly ascending order.

[Copilot]

Pull request overview

This PR extends the MinLZ block search specification to support (1) “index-only” streams that reference blocks in another stream, and (2) entropy-compressed per-block search tables to reduce index size while keeping decoding fast.

Changes:

• Document “Search Index only Streams” that replace block payloads with Remote Block Reference chunks (0x47).
• Add a new “Compressed Block Search Table” chunk (0x46) that stores search-table bits compressed with huff0 and includes a checksum.
• Specify the Remote Block Reference (0x47) payload format, including support for multiple referenced blocks via relative offsets.

Comments suppressed due to low confidence (1)

SPEC_SEARCH.md:152

• This sentence appears to have a grammatical error: "This block replaces a block that and indicates the offset of the block in the stream." Please fix the wording (e.g., remove the stray "that") and clarify what stream the offset is relative to (the current index-only stream vs the referenced data stream).

This block replaces a block that and indicates the offset of the block in the stream.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

writer.go:1226

• appendSearchTableEitherChunk discards the error from appendSearchTableCompressedChunk. If the 0x46 encoder fails unexpectedly, this will silently emit a 0x45 chunk and lose the error. Consider handling the error (e.g., set writer error state via w.err(err) and return the 0x45 chunk only when err == nil && !ok).

func (w *Writer) appendSearchTableEitherChunk(dst []byte, reductions uint8, table []byte) []byte {
	if w.searchCfg.compression != nil && w.searchCfg.compression.enabled {
		e := cstEncoderPool.Get().(*cstEncoder)
		out, ok, _ := appendSearchTableCompressedChunk(dst, w.searchCfg, reductions, table, e)
		cstEncoderPool.Put(e)

search_compressed.go:157

• CompressedSearchSkipPct's doc comment says the default is 5.0, but WithCompression() initializes skipPctTimes100 from cstDefaultSkipPctTimes100 (currently 1000 = 10.0%), and the CLI default flag is also 10.0. Please align the documentation with the actual default behavior.

// CompressedSearchSkipPct sets the popcount-band half-width (in percent) below
// which compression is skipped. The default is 5.0.
func CompressedSearchSkipPct(pct float64) CompressedSearchOption {