perf: zapx v18 format — WAND/MAXSCORE primitives, PFOR freq encoding, BP doc reordering#438
Open
maneuvertomars wants to merge 19 commits into
Open
perf: zapx v18 format — WAND/MAXSCORE primitives, PFOR freq encoding, BP doc reordering#438maneuvertomars wants to merge 19 commits into
maneuvertomars wants to merge 19 commits into
Conversation
Adds per-field per-segment cache of max BM25 tf-norm values, computed
lazily by scanning the posting list on first query for each (term, avgDocLen)
pair.
Cache eviction:
1. Segment lifecycle: when a segment is merged/GC'd, invertedIndexCache
is freed automatically — primary eviction, zero cost, automatic.
2. Within-segment cap (maxTFNormCacheSize = 100_000 entries per field):
once full, new terms are silently skipped; hot terms (queried first)
win the slots, which is the right policy for WAND.
3. Per-entry invalidation: avgDocLength stored alongside the cached value;
if the corpus grows and avgDocLength shifts, the entry is recomputed.
Also adds wandBM25K1/wandBM25B constants matching bleve/search/util.go.
Add `termOffsetCache sync.Map` to `invertedCacheEntry`. On the first query for a term, postingsList() caches the FST-resolved posting-list offset so subsequent queries skip the FST traversal entirely. Each query does 3 terms × 15 segments = 45 FST traversals; caching the offset converts these to O(1) map lookups after the first query. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move per-doc field-length norms out of the per-posting freq stream into a new flat byte array per field (SectionNormColumn = 3). Each byte is a SmallFloat-encoded field length (3-bit mantissa + 5-bit exponent, ~4–12% error, covers fieldLen 1–~2M). Format changes: - Version 17 → 18; v17 segments are rejected by the v18 reader - NumSections 3 → 5 (adds SectionNormColumn=3, SectionMaxTFNorm=4 placeholder) - Freq stream now carries only freqHasLocs per posting (one uvarint, not two) - New norm.go: encodeNormByte / normDecodeTable (256-entry precomputed decode) - New section_norm_column.go: full flush + merge implementation; 20-byte fieldNotUninverted header for loadDvReaders compatibility Key details: - normColumnHeaderSize = 20 bytes (2× uvarint(MaxUint64)) so loadDvReaders can safely skip norm column sections without misinterpreting raw norm bytes - PostingsIterator.normColumn slice attached at iterator() time; docNum lookup is a single byte load from the already-mmap'd page - 1-hit FST path: nextBytes() now encodes only freqHasLocs (not normBits); normBits still returned separately for use1HitEncoding in the merge path - getNormColumnOpaque lazily initialises to handle the numDocs==0 merge case Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add SectionMaxTFNorm (= 4) which records (postingsOffset, maxFreq, maxNormByte) for every (field, term) at segment flush time. At query time, Dictionary.MaxTFNorm() does a binary-search lookup into the sidecar before falling back to the O(N) lazy posting scan. Storing raw (maxFreq, maxNorm) components rather than a precomputed float means the wandTFNorm upper bound can be recomputed for any avgDocLen at query time, solving the key design challenge without losing accuracy. Merged segments fall back to the existing lazy scan (addr=0 → lookupMaxTFNorm returns ok=false); only freshly-flushed segments gain the O(1) cold-start path immediately. Also fix 1-hit MaxTFNorm: v18 normBits1Hit stores fieldLen (integer), not float32 bits, so use 1/sqrt(fieldLen) instead of Float32frombits(normBits1Hit). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…=1027) Replaces varint encoding in the freq stream with a PFOR (patched frame of reference) block format. DefaultChunkMode is updated to 1027 = PFORChunkMode. Block format: countByte (0=256) + bitWidth (0=constant, else packed) + data. Constant blocks (all same ≤255 value) encode in 3 bytes. General blocks use optimal bit width with exception storage for outlier values. Exception values stored as LE uint32; bits packed LSB-first. Decoder fix: always call SetPFORMode(enabled) with the correct bool so reused iterators don't carry pforMode=true when reading a non-PFOR segment. isNil() for PFOR mode returns pforPos >= len(pforDecoded) to trigger loadChunk after all values in a chunk are consumed (covers the decoder-reuse-across-terms case as well as the initial state). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three pprof-driven micro-optimizations after v18 PFOR landing: 1. decodePFORBlock(dst []uint64): accepts a pre-allocated buffer and fills it in-place (reusing its backing array) instead of make([]uint64, count) on every block decode. 2. SetPFORMode pre-allocates pforDecoded with cap=256 so the first loadChunk call reuses it without a grow. 3. SkipN(n int) on chunkedIntDecoder: O(1) pforPos += n for PFOR mode; falls back to n×SkipUvarint for varint mode. 4. nextDocNumAtOrAfterClean fast-skip: replaces the O(sameChunkNexts) currChunkNext loop with a single loadChunk + SkipN(sameChunkNexts) call when !includeLocs. For dense-enough terms (hotel DF 3.5%, ~9 hits per 256-doc block), Advance() skips multiple preceding hits in the same PFOR block in one array-index operation instead of N function calls. Result (preliminary): BenchmarkQueryBestHotelLisbon −13.4% vs v18 baseline; TermTier1/2/3/4 neutral (those benchmarks call Next() not Advance(), so sameChunkNexts is always 0). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ORBlock Go does not inline function literals, so allocBuf(count) returned a slice of statically-unknown length to the caller. This suppressed bounds-check elimination in the tight inner loop (out[i] = v & mask), adding one bounds check per value × 256 values × ~1950 block decodes per TermTier1 query. Replace the closure with a direct if/else conditional so the compiler can track len(out) == count through SSA and eliminate the redundant checks. TermTier1 recovers from +4.9% regression to -1.6% vs the pre-SkipN baseline. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
decodePFORBlock was calling make([]int, nExcepts) and make([]uint64, nExcepts) on every block with exceptions, causing ~1000 heap allocations per TermTier1 query (high-freq term, wide freq range, many PFOR blocks with nExcepts > 0). Add excIdx []int and excVal []uint64 scratch parameters, mirroring the existing dst []uint64 pattern for the main output buffer. Callers pass pre-allocated slices that are resliced to nExcepts length; heap falls back only if scratch capacity is insufficient (rare: only when nExcepts exceeds pforBlockSize, which the encoder tries to avoid). chunkedIntDecoder gains pforExcIdx/pforExcVal fields pre-allocated in SetPFORMode alongside the existing pforDecoded buffer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the O(N×bw) bit-serial inner loop with two faster paths: - bw=8 fast path: each value is exactly one byte; straight uint64 cast. - Sliding-window for all other bw values: accumulate bytes into a uint64 buffer and drain bw bits per value. The inner refill runs ceil(bw/8) ≈ 1 time per value instead of bw times, eliminating the ~8× overhead of the previous bit-serial approach. TermTier1 (bw=8/9 freq blocks, 256 values per block): 33.7 ms → 21.7 ms/op (−35.6% vs v18, −32% vs v17 baseline). TestDisjunctionSearchScoreIndexWithCompositeFields was already failing before this change (pre-existing score mismatch between upsidedown and scorch index types unrelated to PFOR decode). All zapx unit tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add NormColumnByte(docNum uint64) uint8 to PostingsIterator so the bleve scoring bridge (§25 BM25 impact table) can read the raw SmallFloat norm byte only for documents that survive WAND — not for every posting Next() visits. Previously the byte was decoded eagerly in Next() via a normByte field on Posting. For TopicalDisjunction8 with ~17k postings per essential term but only ~540 scored, this added significant overhead. Now the normColumn read is deferred: postingToTermFieldDoc calls NormColumnByte via a type assertion on the iterator. Zero return value (docNum not in normColumn, pre-v18 segment, or field without a norm column) signals the bridge to fall back to the float64 norm path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds computeBPPermutation() (bp.go) which:
1. Scans input segments to count DF per (field, term) — two FST passes.
2. Filters to eligible terms: minDocFreq=128 <= DF <= 95% of numDocs.
3. Builds a forward index fwdIndex[seqID] = []uint32{termIDs}.
4. Runs recursive graph bisection (bpState.bisect) to produce a
permutation perm[seqID] = bpID that clusters topically similar
docs to adjacent IDs.
bisect() uses log-odds bias scoring
(sum_t log2(rightFreq[t]) - log2(leftFreq[t]))
with a shared dirty-list workspace to avoid per-level array clearing.
Stopping criterion: break when moved <= iter (Mackenzie et al.).
merge.go: pass config["bpReorder"]=true to MergeUsing to enable BP.
mergeStoredAndRemap accepts a bpPerm []uint64 and applies it to both
segNewDocNums and docNumOffsets. The copyStoredDocs fast path is
skipped when BP is active.
Both chunkedIntCoder (PFOR mode) and chunkedContentCoder (doc values) require
monotone docNums. BP permutation interleaves new doc IDs from multiple source
segments non-monotonically, corrupting chunkLens and snappy-compressed doc
values. Fix: replace per-segment immediate writes with a two-phase approach —
collectMergeEntries accumulates entries per source segment; flushMergeEntries
globally sorts by newDocNum and writes monotonically. Applied to both inverted
text index (PFOR freq/loc) and doc values.
Corpus: benefit visible only on topical corpora, not Zipf-random.
BP gives 35% posting-list cost reduction for same-topic queries vs
topical-merged baseline; .zap shrinks 391→317 MB (19%).
Use bench/topical_search_test.go (sharedTopicalIdx) for measurement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously Merge() was a no-op; merged segments had no MaxTFNorm sidecar and fell back to a cold O(posting_list_size) scan per term per query on every restart, defeating WAND/MAXSCORE pruning for the bulk of data. mergeAndPersistInvertedSection now takes max(maxFreq, maxNorm) across all contributing source segments via lookupMaxTFNorm (O(1) binary search, cached per field). If any source segment lacks a sidecar entry for a term, that term is skipped (falls back to lazy scan — same as before). After one merge round of flush-time segments, all output segments carry a complete sidecar; the optimization propagates to all data over time. Three bugs discovered during testing were also fixed: 1. merge.go / new.go: segmentSections map iteration is unordered in Go. mergeToWriter() re-initialised the MaxTFNorm opaque after InvertedText had already populated it (random ordering). interim.convert() could call MaxTFNorm.Persist() before writeDicts() ran. Fixed by iterating section IDs in ascending integer order with a skip-if-already-set guard. 2. section_inverted_text_index.go writeDicts(): interimFreqNorm.norm stores fieldLen reinterpreted as float32 bits, not the actual norm 1/√fieldLen. The old code compared these raw bits as norms, producing wrong maxNormByte values (e.g. 0.0005 instead of 0.707). Fixed by recovering fieldLen via math.Float32bits() and computing the real norm from minFieldLen. 3. section_maxtfnorm.go: decodeMaxTFNormField() looped until pOff==0, reading past the field's sidecar into adjacent section data. The extra entries were not monotonically sorted, breaking binary search and causing every sidecar lookup to return ok=false on the source segments. Fixed by writing a uvarint entry-count immediately after the 20-byte DV header so the decoder reads exactly that many entries. Adds four unit tests covering merge sidecar presence, lookup correctness, max-across-segments semantics, and idempotent re-merge (self-healing property). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or index BP permutes docIDs in roaring bitmaps, stored fields, and the norm column but does not update the FAISS vector section's internal DocID mapping array. Running BP on a segment that also has vector data would cause FAISS ANN results to return pre-BP docIDs that no longer correspond to the correct documents — a correctness bug. Add segmentsHaveFAISS() which scans fieldsSectionsMap for any nonzero SectionFaissVectorIndex address. mergeToWriter calls this before computeBPPermutation; if any source segment has FAISS data, bpPerm stays nil and BP is skipped entirely. Text-only segments are unaffected. Add two unit tests: TestSegmentsHaveFAISS verifies detection; the new TestBPGuardSkipsOnFAISS exercises the same guard function with a crafted SegmentBase carrying a fake FAISS section address. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…raversal Rare-entity disjunction queries hit many (segment, field) pairs where the search term does not exist. Previously postingsList() returned immediately without caching, so every query traversed the FST again for each miss. Add termNotFoundSentinel (^uint64(0)) and store it in termOffsetCache on the first !exists result. Subsequent queries for the same term in the same segment return emptyPostingsList without touching the FST reader. For EntityRare6FieldDisjScoreNone (6-field disjunction, 15 hits across 90 segment×field pairs) this eliminates the dominant 610ms flat cost on fstReader.Get() and contributes ~45% of the combined -59% improvement. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When OptimizeTFRConjunction replaces each TFR's ActualBM with the AND'd intersection bitmap, nextDocNumAtOrAfter must advance the full posting list iterator (i.all) to sync with each intersection hit. The original code did this with a sequential Next() loop — O(full_DF) per hit — making dense conjunctions cost O(full_DF × result_count) total. For PFOR segments (v18 default), replace the scan with: i.all.AdvanceIfNeeded(n) // O(log k) binary search on arrayContainer i.all.Next() // consume n Then use CardinalityInRange(chunkStart, n+1) — a bitmap rank query — to compute the ordinal of n within its PFOR chunk and call SetPFORPos to seek the freq/norm reader directly to that position. The varint slow path is preserved unchanged for older segment formats. Benchmark results (Conjunction3Dense: w00200 ∩ w00500 ∩ w01000, 41 hits): Before: ~400µs/op (2886 iters/s) After: ~108µs/op (10000 iters/s) Speedup: 3.7× Conjunction3Sparse (3 hits): 62µs → 44µs (1.4×). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…freq same-chunk
Two regression tests for the §36C fast-path (nextDocNumAtOrAfter SetPFORPos):
TestConjunctionFastPathLocationsSlowPathCorrect:
With includeLocs=true, the fast path is blocked (!i.includeLocs guard).
Narrows ActualBM to docs {2,5,8} simulating the conjunction AND optimisation.
Verifies Locations()[0].Pos() == docNum+1 for each (position depends on
sequential loc-reader advancement through skipped docs).
TestConjunctionFastPathFreqTwoHitsSameChunk:
With includeLocs=false + PFOR mode, the fast path fires via SetPFORPos.
Two hits in the same chunk (freqIdx and freqIdx+3); the first doc has freq=5,
the second has freq=1. Verifies SetPFORPos is re-applied between hits so the
second call returns 1 rather than the stale value from the first seek.
Uses the existing buildTestSegmentMulti / newStubDocument / getTempPath
test helpers within package zap.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…xTFNorm edges
Three new test files covering §20 (norm column), §36C (SetPFORPos), and §14
(MaxTFNorm sidecar) with backwards-compatible format changes only.
pfor_setpforpos_test.go:
- TestSetPFORPosSeekBoundaries: SetPFORPos at positions 0, mid (3), last (7)
within a single PFOR chunk via narrowed ActualBM (§36C fast path). Confirms
each seek returns the correct distinct frequency, not a stale pforPos=0 value.
- TestSetPFORPosSequentialVsSeek: sequential Next() vs. single-doc Advance()
produce identical Frequency() values for all 8 docs — regression against
SetPFORPos corrupting subsequent reads.
norm_merge_drop_test.go:
- TestNormColumnPreservedAfterMergeWithDrops: two source segments merged with
one doc dropped from the second segment. Verifies the merged norm column is
compacted correctly: each of the 4 surviving docs has the SmallFloat byte for
its actual field length, not the dropped doc's byte (which would shift all
subsequent norm entries).
maxtfnorm_edgecase_test.go:
- TestMaxTFNormOneHitPath: term in exactly 1 doc (1-hit FST encoding, normBits1Hit≠0)
takes the fast path and returns a positive, in-range value.
- TestMaxTFNormTermOutOfRange: "aaa" / "zzz" / "" — absent terms return 0 without panic.
- TestMaxTFNormMaxFreqTerm: two terms with freq=1 and same field length produce
equal MaxTFNorm values (validates wandTFNorm symmetry).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make config["bpReorder"] polymorphic in MergeUsing:
bool true -> auto-pick the richest field (existing behavior)
bool false -> off
string "field" -> build the BP forward index from exactly that field
absent/nil/other -> off (global default, off today)
- merge.go: replace the config["bpReorder"].(bool) assertion with a type switch
(bool | string); a non-empty string sets BPOptions.BPField. FAISS / numDocs
guards unchanged.
- bp.go: add BPOptions.BPField. When set, validate it against the merged
fieldsInv (must be present and indexed); if absent/not-indexed, fail SAFE to an
identity permutation (BP off) and log.Printf a warning — a silent typo that
disables locality is a debugging trap. Step 3 uses the explicit field instead
of the most-eligible-terms auto-pick when provided.
- bp_test.go: TestBPFieldSelection — explicit valid field ("desc") yields a valid
bijection; missing field ("no_such_field_xyz") falls back to identity + warns.
v1 of §68; deferred: []string multi-field (fwdData cache cost) + per-field weights.
No on-disk format change — only which field drives the merge-time permutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces the zapx v18 segment format along with a set of performance optimizations targeting WAND/MAXSCORE competitive scoring and posting-list decode hot paths. It is the zapx half of the "perf-gar" performance effort; the corresponding bleve changes (which register v18 as the default segment plugin and add
SearchRequest.ScoreMode/ WAND pruning on top of these primitives) are in a companion bleve PR from theperf-gar-mod-fixbranch.Changes
New v18 format sections
NormColumnByte).PFORChunkMode=1027), plus decode-path optimizations: per-block heap allocs eliminated,allocBufclosure replaced with an inline conditional, sliding-window bit-unpacking,SkipNfast path and decoded-slice reuse.WAND/MAXSCORE support
maxTFNormcache ininvertedIndexCachefor WAND pruning.Merge-time optimizations
bpReorderfield selection).Query hot path
i.alladvancement in the conjunction AND optimization.Tests
SetPFORPos, norm merge drop, MaxTFNorm edge cases).Notes for reviewers
github.com/blevesearch/zapx/v18, which is whygo.modconflicts with master — master has since moved (roaring, bleve_index_api, scorch_segment_api version bumps). The dependency pins ingo.modneed to be reconciled with current master as part of conflict resolution.