Skip to content

Add Sidecar Search Indexes#42

Merged
klauspost merged 16 commits into
minio:mainfrom
klauspost:sidecar-search
Jun 9, 2026
Merged

Add Sidecar Search Indexes#42
klauspost merged 16 commits into
minio:mainfrom
klauspost:sidecar-search

Conversation

@klauspost

@klauspost klauspost commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Sidecar Streams

Search tables normally live inside the compressed stream — one per block,
right in front of the corresponding data. A sidecar stream instead puts
every search table into a separate file, leaving the main .mz stream as
plain compressed data. Searching then takes two inputs: the sidecar (read
once, top-to-bottom) and the main stream (random-access via io.ReaderAt,
typically a regular file).

Why?

  • You can't (or don't want to) re-encode the data. Maybe the .mz was
    written long ago, or it lives on read-only storage. BuildSidecar decodes
    each block, builds fresh tables, and writes the sidecar — the original
    .mz is never touched.
  • You want to store the index separately. Indexes can be regenerated;
    data often can't. Putting them on different storage tiers (or even
    different hosts) is a clean way to express that.
  • You want to drop the indexes from an existing indexed stream. If you
    compressed with -search but later want to slim the main file without
    losing search, extract the index to a sidecar and strip the main.
  • You want multiple search configurations for the same data. The inline
    writer only carries one config; a sidecar can carry many (for example one
    tuned for short patterns and one tuned for JSON keys).

The main stream and the sidecar are both valid MinLZ streams on their
own
. A regular NewReader over a sidecar produces zero data bytes (only
skippable chunks); over a stripped main stream it produces the original
payload.

Sidecar files conventionally use the .mzs extension (sits next to .mz),
but they are ordinary MinLZ streams and can have any filename.

Three ways to produce a sidecar

1. While compressing — WriterSidecar

The main writer behaves normally; the search-table chunks are diverted to a
second io.Writer:

mainF, _ := os.Create("foo.mz")
sideF, _ := os.Create("foo.mz.mzs")
cfg := minlz.NewSearchTableConfig().WithMatchLen(6)
w := minlz.NewWriter(mainF,
    minlz.WriterSearchTable(cfg),
    minlz.WriterSidecar(sideF),
)
w.Write(data)
w.Close()

2. From an existing stream that has no indexes — BuildSidecar

Reads the source once, decompresses each block, builds fresh tables, and
writes the sidecar. The source is not modified:

src, _ := os.Open("foo.mz")
side, _ := os.Create("foo.mz.mzs")
err := minlz.BuildSidecar(side, src,
    minlz.SidecarSearchTable(minlz.NewSearchTableConfig().WithMatchLen(6)),
)

Pass SidecarSearchTable multiple times to embed several configs in one
sidecar. At search time a block is skipped if any embedded table proves
the pattern absent; tables that don't apply to the query are ignored.

3. From an existing indexed stream — ExtractSidecar

Copies the existing 0x44/0x45/0x46 chunks verbatim into a sidecar. No
decoding required. Two modes:

// (a) No stripping. Sidecar offsets reference src's original layout, so
//     the original file must be kept and searched against.
minlz.ExtractSidecar(sideOut, nil, src)

// (b) With stripping. Re-emits src to newMainOut without the search chunks;
//     sidecar offsets reference the new layout. A fresh seek index is
//     appended to newMainOut so it remains seekable on its own.
minlz.ExtractSidecar(sideOut, newMainOut, src)

Searching with a sidecar

mainF, _ := os.Open("foo.mz")        // io.ReaderAt
sideF, _ := os.Open("foo.mz.mzs")    // io.Reader

searcher := minlz.NewSidecarSearcher(mainF, sideF)
searcher.Search([]byte("pattern"), func(r minlz.SearchResult) error {
    fmt.Printf("match at stream offset %d\n", r.StreamOffset)
    return nil
})

The searcher walks the sidecar once and only touches the main stream for
blocks that aren't proven absent. Adjacent must-decode blocks are coalesced
into a single ReadAt; skipped blocks issue no I/O against the main stream
until the caller invokes result.PrevBlock() for boundary context (then
they are fetched and decoded on demand).

SidecarSearcher accepts the same BlockSearchOptions as BlockSearcher
(BlockSearchBailOnMissing, BlockSearchCollectStats,
BlockSearchIgnoreCRC, …). Multiple goroutines may run independent
SidecarSearchers against the same main file concurrently — each owns its
own sidecar io.Reader and the underlying ReadAt is shared safely.

Commandline

mz sidecar build   [-search.lens=4,6] [-search.compress] foo.mz   # -> foo.mz.mzs
mz sidecar extract [-newstream stripped.mz]              foo.mz   # -> foo.mz.mzs (+ stripped.mz)
mz search --sidecar foo.mz.mzs  "pattern"  foo.mz                 # search via sidecar

mz sidecar build accepts most of the same -search.* flags as
mz c -search (match length, prefixes, compression, population limits).
The -search.lens flag is comma-separated to request multiple configs in
one sidecar.

Sidecar format at a glance

A sidecar stream contains, in order:

  1. Stream identifier (0xff) — same maximum block size as the main stream.
  2. One or more Search Table Info chunks (0x44) — one per embedded config.
  3. For each data block in the main stream:
    • Zero or more Search Table chunks (0x45/0x46) — one per config
      that produced a usable table for that block.
    • One Remote Block Reference chunk (0x47) carrying the block's
      offset in the main stream and its uncompressed size.
  4. EOF chunk (0x20).

For format details see SPEC_SEARCH.md §1.1 / §2.3.

Summary by CodeRabbit

  • New Features

    • Sidecar support: build/extract sidecar index files, new mz sidecar command, automatic .mzs detection, and sidecar-based searching for low‑I/O pattern queries; writer can emit a separate sidecar stream.
  • Improvements

    • Long‑prefix “extras” to index multiple match windows for more accurate multi‑byte prefixes.
    • Search-table compression enabled by default with an explicit disable option and richer compressed‑subblock stats.
    • New CLI flags to control search-table behavior (prefixes/extras/lim/uncompressed/sidecar).
  • Documentation

    • Expanded SEARCH.md and SPEC updates; README ports table now includes a Rust row and clarified feature matrix.

klauspost added 2 commits June 1, 2026 11:22
Add more block info to references.
Comment thread cmd/mz/compress.go Fixed
Comment thread cmd/mz/compress.go Dismissed
Comment thread cmd/mz/compress.go Dismissed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces sidecar search-index streams for MinLZ, allowing search tables to be stored in a separate .mzs stream while keeping the main .mz stream as plain compressed data. It adds library support for generating/consuming sidecars, updates the search-table format/docs for remote block references (0x47), and extends the mz CLI to build/extract sidecars and to search using them.

Changes:

  • Add sidecar stream generation (WriterSidecar, SetSidecar), sidecar building/extraction (BuildSidecar, ExtractSidecar), and sidecar-based searching (SidecarSearcher).
  • Make per-block search-table compression the default (0x46 emitted when beneficial), plus CLI/documentation updates to reflect new defaults and flags.
  • Add a new mz sidecar {build,extract} CLI and extend mz search with optional sidecar usage / auto-detection.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
writer.go Adds sidecar emission paths (0x44/0x45/0x46 diverted + 0x47 refs) and public WriterSidecar/SetSidecar.
sidecar.go Implements BuildSidecar and ExtractSidecar, including parsing/writing 0x47 remote references.
sidecar_search.go Adds SidecarSearcher that walks the sidecar sequentially and reads main blocks via io.ReaderAt.
sidecar_test.go Comprehensive tests for sidecar generation/searching, concurrency, concat streams, stats parity.
search_table.go Makes table compression default; adds WithoutCompression; adds prefix-driven default tightening.
search_reader.go Extends lazy prev-block support to fetch from io.ReaderAt for sidecar searching.
search_examples_test.go Adds examples for WriterSidecar/BuildSidecar/ExtractSidecar/SidecarSearcher.
cmd/mz/sidecar.go New CLI subcommand for sidecar build/extract.
cmd/mz/search.go Adds --sidecar and sidecar auto-detection; switches to sidecar search when present.
cmd/mz/compress.go Adds -search.sidecar output mode and refactors stream processing options.
minlz.go Adds chunk type constant for 0x47.
SPEC.md / SPEC_SEARCH.md / SEARCH.md / README.md / search_test.go Spec/doc/test updates for 0x46 default compression + 0x47 sidecar references.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread sidecar_search.go Outdated
Comment thread sidecar_search.go
Comment thread sidecar_search.go
Comment thread writer.go Outdated
Comment thread writer.go
Comment thread sidecar.go Outdated
Comment thread sidecar_test.go Outdated
Comment thread writer.go Outdated
Comment thread writer.go
@klauspost klauspost marked this pull request as ready for review June 2, 2026 09:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comment thread writer.go
Comment thread sidecar_search.go

@krisis krisis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 44 changed files in this pull request and generated 4 comments.

Comment thread sidecar.go
Comment thread sidecar.go Outdated
Comment thread SPEC_SEARCH.md Outdated
Comment thread writer.go
@klauspost

Copy link
Copy Markdown
Collaborator Author

Added Extras to long prefixes:

Extras

By default a long-prefix table records one matchLen-byte hash per prefix occurrence.
WithExtras(E) widens that to E+1 overlapping matchLen-byte windows at offsets
0..E after each prefix, and the searcher checks the same E+1 windows per occurrence
in the query. The constraint is matchLen + E ≤ 16, so the post-prefix region the
encoder hashes never exceeds 16 bytes.

cfg := minlz.NewSearchTableConfig().
    WithMatchLen(8).
    WithLongPrefix([]byte(`"timestamp":`)).
    WithExtras(8) // 9 hashes per occurrence (matchLen+extras = 16)

Extras effectively turn the table into a k=E+1 Bloom filter for the post-prefix bytes:
the false-positive probability per occurrence drops from p to p^(E+1). The cost is
proportionally more bits set per occurrence — typical sidecars grow by roughly the same
factor before sparse/huff0 compression kicks in.

When to use it:

  • The prefix fires sparsely (e.g. one structured field per log line) and queries reliably
    carry enough trailing bytes (L ≥ pl + matchLen + E).
  • The single-hash table is collision-bound — many candidate blocks turn out to be false
    positives that have to be decoded.

When to avoid it:

  • The prefix is dense and the table is already population-bound; extras multiply the
    population without adding filtering power.
  • Queries are short and would not fill matchLen + E bytes after the prefix; those
    occurrences become unusable and the searcher falls back.

@klauspost

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2d3782b3-dac2-4027-b696-114ed6c1eb4f

📥 Commits

Reviewing files that changed from the base of the PR and between f38346d and 06b2db8.

📒 Files selected for processing (2)
  • search_compressed.go
  • search_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • search_test.go

📝 Walkthrough

Walkthrough

This PR implements MinLZ sidecar search-index functionality, long-prefix "extras" (E extra match windows), compressed search-table support (0x46), and remote block references (0x47). It adds a reference encoder/decoder, end-to-end CLI wiring (build/extract/search/compress), writer-sidecar emission, sidecar searcher, extensive tests, and documentation/spec updates.

Changes

Sidecar Search Functionality

Layer / File(s) Summary
Search table config and long-prefix extras
search_table.go
SearchTableConfig gains extras and maxReducedPopPctSet, WithExtras(), WithoutCompression(), defaults resolution, and extras-validated wire-format parsing/serialization.
Search index building with extras
search_index.go
buildSearchTable and buildTablePrefixLong index E+1 overlapping windows per long-prefix occurrence and adjust overlap-tail hashing.
Reference search implementation
internal/reference/search_encoder.go, internal/reference/search_decoder.go, internal/reference/search_test.go
New reference encoder/decoder and tests for 0x44/0x45/0x46/0x47 covering bitmap build, compression sub-block encoding, CRC checks, and Contains semantics.
Search reader with sidecar and compressed support
search_reader.go
Adds remote/sidecar lazy decoding, blockTableUnusable path, new patternCanUseConfig helper, extras-aware matching/deferral, and compressed-table stats fields.
Compressed search table encoding/decoding
search_compressed.go
Refactors CST decoder with reusable cstJob jobs, runJob() worker, header-only parsing, and sub-block stats SubBlockSize/SubBlocks.
Writer sidecar support
writer.go
New WriterSidecar option and SetSidecar; writer emits search chunks into sidecar and writes 0x47 remote refs into main stream; manages header/init/EOF lifecycle.
Sidecar building and extraction
sidecar.go
Implements BuildSidecar to rebuild search chunks and emit 0x47 refs, and ExtractSidecar to copy/extract search chunks and optionally produce a stripped main stream.
Sidecar pattern search
sidecar_search.go
SidecarSearcher reads sidecar streams, batches refs, coalesces ReadAt, skips/decodes blocks via multi-table policy, supports deferred boundary matches, and reports stats.
CLI sidecar commands
cmd/mz/sidecar.go, cmd/mz/compress.go, cmd/mz/search.go, cmd/mz/main.go
Adds `mz sidecar build
Sidecar and search tests
sidecar_test.go, search_test.go
Comprehensive tests for writer/build/extract/search parity, extras semantics, compressed-table behavior, remote ref round-trips, deferred decoding, and benchmarks.

Documentation and Code Cleanup

Layer / File(s) Summary
Specification updates
SPEC_SEARCH.md, SPEC.md
Spec updated to define sidecar index streams, 0x47 remote block reference format, long-prefix extras and appendix updates for lookup/encoder overlap.
User documentation
README.md, SEARCH.md
README: ports table updated (adds Rust row). SEARCH.md: table-reduction, prefixes/extras, compressed-by-default framing, sidecar streams section, CLI/API examples and flags.
Formatting and small refactors
many files (_generate/gen.go, tests, helpers, examples)
Non-functional changes: octal literal style, var→:= refactors, comment spacing, example output formatting, and minor grouping/whitespace edits.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

🐰 With sidecars packed and searches swift,
Extras dance where prefixes drift,
Blocks may roam while tables stay,
Lazy decode saves the day!

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 17

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
cmd/mz/search.go (1)

67-79: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Reject an explicit --sidecar when more than one input/glob match is searched.

The same sidecar path is forwarded into every searchFile call here. With multiple matched inputs, later files will be searched against an unrelated index, which can silently produce false positives/negatives instead of failing fast.

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/mz/compress.go`:
- Around line 224-239: The buffered writers (bw/sbw) and output files
(dstFile/sideFile) can report errors on Flush() and Close() that are currently
ignored, so update the compress command around wr.Reset/bw and wr.SetSidecar/sbw
to check and propagate errors from sbw.Flush(), bw.Flush(), sideFile.Close(),
and dstFile.Close(); ensure wr.Close() errors are handled too. Concretely, after
calling wr.Close() call and check bw.Flush() and sbw.Flush() (if used) and then
close the underlying files (dstFile.Close(), sideFile.Close()) and pass any
non-nil errors to exitErr (or return them) so disk-full or final short-write
failures are reported; reference functions/vars: wr.Reset, wr.Close,
wr.SetSidecar, bw, sbw, dstFile, sideFile, and exitErr.

In `@cmd/mz/sidecar.go`:
- Around line 104-108: Before opening/truncating the destination, detect and
reject cases where the destination aliases the input: resolve both src and dst
to their absolute/canonical paths (e.g., filepath.Abs and/or
filepath.EvalSymlinks) and compare them, and if they are equal return an error
instead of proceeding; apply the same check in the second block (the code around
the 175-186 range). Insert this validation before any file is opened/truncated
(i.e., before the code that uses dst and before calling BuildSidecar) and use
the existing variables src, dst, outFile, minlzSidecarExt and the BuildSidecar
call to locate where to add the check.
- Around line 239-268: The code opens the input then creates/truncates outputs
(sideOut/newOut) which can clobber the source or alias outputs; before opening
any files (i.e., before os.Open(src) and before creating sideOut/newOut) resolve
and compare the canonical paths for src, dst (derived from *outFile or
minlzSidecarExt) and *newStream using filepath.Abs and filepath.EvalSymlinks,
and if any two canonical paths are equal return an error via exitErr (e.g.,
"destination X conflicts with source Y" or "outputs X and Y alias each other");
update the logic around variables src, dst, *outFile, *newStream, sideOut and
newOut (and any ExtractSidecar invocation) so path-conflict checks run and fail
fast prior to opening/truncating files.

In `@internal/reference/search_decoder.go`:
- Around line 186-214: ParseRemoteBlockRefChunk currently accepts an empty 0x47
payload and returns (nil, nil); change it to reject empty payloads by adding an
explicit check at the start of ParseRemoteBlockRefChunk that if len(payload) ==
0 it returns (nil, fmt.Errorf("minlz: 0x47 empty payload")) so behavior matches
sidecar.go and other uvarint error patterns.
- Around line 95-104: The parser currently allows tc and nBlocks values beyond
production limits; add validation to mirror the production 0x46 header checks by
returning an error when tc > 16 and when nBlocks > 16. Locate the parsing
routine that computes log2bs, expectedSize, blockSize and nBlocks (using
variables log2bs, expectedSize, blockSize, nBlocks and the header field tc) and
insert guards that reject (fmt.Errorf) any tc greater than 16 and any computed
nBlocks greater than 16, consistent with the existing error style used for
invalid huff0 block sizes.

In `@minlz_test.go`:
- Around line 36-39: Remove the unused constant maxUint and adjust the
declaration of maxInt accordingly: delete maxUint and define maxInt using the
expression previously derived from maxUint (int(^uint(0) >> 1)) or inline the
numeric value where used; update the const block containing maxInt so there are
no unused constants (references: maxUint, maxInt).
- Around line 1753-1755: The example test in minlz_test.go uses the
non-canonical marker "// OUTPUT:" so Go's example runner won't match the
expected stdout; change the marker to the canonical "// Output:" (capitalize
only the O) for the example block that currently shows "// OUTPUT:" and the two
lines "Callback: Chunk Custom Data <nil>" and "Stream data: some data" so the
example test will be recognized and validated by go test.

In `@README.md`:
- Around line 914-918: Update the paragraph that currently claims there are "no
ports" to accurately reflect the new table entries by replacing that statement
with a concise sentence noting that official Go and Rust ports exist and an
experimental C port/Gist is available (or alternatively remove the absolute "no
ports" claim); locate the surrounding text in README.md that asserts "no ports"
and modify it to say something like "There are Go and Rust ports (and an
experimental C gist) — see the table below" so the prose aligns with the table
entries.

In `@search_compressed_test.go`:
- Around line 683-685: The test's sub-block sum omits the new BlocksSparse field
from CompressedSearchStats; update the sum calculation (totalBlocks) to include
got.BlocksSparse and adjust the failure message if desired so the assertion
compares got.SubBlocks against the sum that now includes BlocksSparse
(referencing totalBlocks, got.BlocksOwnTable, got.BlocksGlobalTable,
got.BlocksRaw, got.BlocksRLE, and got.BlocksSparse).

In `@search_compressed.go`:
- Around line 844-859: The fast-path parser parseSearchTableCompressedHeader
currently skips validating the fixed 0x46 chunk header; update it to mirror the
lightweight structural checks in parseSearchTableCompressed by verifying the
payload is long enough for the header byte and that payload[0] == 0x46 (or the
same header constant used in parseSearchTableCompressed), and return an error if
the header byte is wrong or overall payload length is insufficient before
reading reductions; keep the existing parseSearchInfo call and prefix-size
offset logic but perform the header check before returning reductions to prevent
malformed chunks from bypassing validation.

In `@search_table.go`:
- Around line 156-158: The WithExtras setter currently casts int to uint8 before
validation causing overflow (e.g., 256 -> 0); fix by validating the input range
in SearchTableConfig.WithExtras (check n >= 0 && n <= 255) before doing c.extras
= uint8(n) and handle out-of-range values explicitly (return unchanged config
and/or return an error/modify the API to propagate errors), and likewise apply
the same pre-cast range check to the other setter(s) that cast an int to uint8
(the related setters referenced around lines 230-237) and make validate() assert
that extras was set within [0,255] to catch any future regressions.

In `@SEARCH.md`:
- Around line 393-397: The fenced code block containing the example mz sidecar
commands (the lines starting with "mz sidecar build", "mz sidecar extract", and
"mz search --sidecar") lacks a language tag; update the opening triple-backtick
to include a language such as bash or text (e.g. ```bash) so linters recognize
it as a shell snippet and avoid markdown lint failures.

In `@sidecar_search.go`:
- Around line 385-395: The code incorrectly assumes len(tables)==1 implies a
single-config stream and sets s.sideDeferred based only on per-block table
count, which causes resolveSideDeferred to compare deferred hashes against
nextTables[0] from a different config; either restrict this optimization to
single-config streams by adding a guard requiring len(s.streamInfos)==1 before
setting s.sideDeferred (referencing tables, s.sideDeferred, patternDeferHashes,
flushBatch, s.sideDeferredHashes, s.sideDeferredBase), or instead attach the
deferred config identity when creating s.sideDeferred (e.g., store the table's
config id) and update resolveSideDeferred to only compare hashes against
nextTables[0] if that table’s config id matches the deferred config (use
resolveSideDeferred, nextTables and the stored deferred config id).
- Around line 198-206: The code currently destructively updates s.maxBlock when
reading the first sidecar header (using streamBlockSizeFromHeaderByte) and never
recomputes it for subsequent ChunkTypeStreamIdentifier headers, causing later
larger streams to be rejected; change the logic to keep the configured ceiling
separate (e.g., preserve the original configured max as s.maxBlockConfigured or
similar), set s.sideMaxBlk from streamBlockSizeFromHeaderByte for each header,
and recompute a per-stream active limit (e.g., activeMax :=
min(s.maxBlockConfigured, s.sideMaxBlk)) whenever you see a
ChunkTypeStreamIdentifier so both validation and prevLazy use this activeMax
instead of mutating s.maxBlock. Ensure updates at the same spots mentioned
(current block and the other occurrence around lines 483-489) so each
concatenated stream recalculates limits on its header.

In `@sidecar_test.go`:
- Around line 567-577: The test currently fails on valid coalesced reads because
it treats any overlap in rr.ranges as an error, but decodeBatch intentionally
reads a conservative [rangeStart, rangeEnd) for the last ref in a batch; update
the check to only fail when one range is fully contained in the previous range
(i.e., ranges[i][0] < ranges[i-1][0] || ranges[i][1] <= ranges[i-1][1]) instead
of any partial overlap, so allow partial overlaps that extend past the previous
end; refer to rr.ranges and decodeBatch when making this change.

In `@SPEC_SEARCH.md`:
- Line 14: The heading "1.1 Sidecar Search Index Streams" is using H3 (###)
after an H1, breaking incremental heading levels; change that header from "###
1.1 Sidecar Search Index Streams" to H2 ("## 1.1 Sidecar Search Index Streams")
so markdownlint-compliant incremental heading levels are preserved.
- Around line 436-446: The fenced pseudocode block lacks a language identifier
which triggers markdownlint; update the fence surrounding the snippet (the block
containing variables like P, pfx, K, M, E and the loop/checked logic) to include
a language tag such as `text` (e.g., change the opening ``` to ```text) so the
block is properly marked as non-executable pseudocode.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 72ba34b9-ab45-42b9-b290-f8fe017efa08

📥 Commits

Reviewing files that changed from the base of the PR and between 9dbed78 and 2508d8c.

📒 Files selected for processing (44)
  • README.md
  • SEARCH.md
  • SPEC.md
  • SPEC_SEARCH.md
  • _generate/gen.go
  • benchmarks_test.go
  • cmd/internal/filepathx/filepathx.go
  • cmd/internal/filepathx/filepathx_test.go
  • cmd/internal/readahead/reader.go
  • cmd/internal/shttp/seekinghttp.go
  • cmd/mz/compress.go
  • cmd/mz/main.go
  • cmd/mz/search.go
  • cmd/mz/sidecar.go
  • decode.go
  • encode.go
  • encode_l3.go
  • encode_test.go
  • fuzz_test.go
  • index.go
  • index_test.go
  • internal/fuzz/helpers.go
  • internal/reference/decoder.go
  • internal/reference/search_decoder.go
  • internal/reference/search_encoder.go
  • internal/reference/search_test.go
  • internal/reference/stream.go
  • lz4convert.go
  • lz4convert_test.go
  • minlz.go
  • minlz_test.go
  • search_compressed.go
  • search_compressed_test.go
  • search_examples_test.go
  • search_index.go
  • search_reader.go
  • search_table.go
  • search_test.go
  • sidecar.go
  • sidecar_search.go
  • sidecar_test.go
  • unsafe_enabled.go
  • writer.go
  • writer_test.go

Comment thread cmd/mz/compress.go
Comment thread cmd/mz/sidecar.go
Comment thread cmd/mz/sidecar.go
Comment thread internal/reference/search_decoder.go
Comment thread internal/reference/search_decoder.go
Comment thread sidecar_search.go Outdated
Comment thread sidecar_search.go Outdated
Comment thread sidecar_test.go
Comment thread SPEC_SEARCH.md Outdated
Comment thread SPEC_SEARCH.md Outdated
klauspost added 2 commits June 9, 2026 11:31
…eductions byte only) was weaker than the full parser's (off+1+4+2 > len(payload), room for the whole fixed header: reductions + crc + log2bs + tc). I tightened the header parser to mirror the full one exactly.
@klauspost klauspost merged commit 828e78a into minio:main Jun 9, 2026
59 checks passed
@klauspost klauspost deleted the sidecar-search branch June 9, 2026 14:41
@coderabbitai coderabbitai Bot mentioned this pull request Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants