Skip to content

perf(fts): bound classic conjunction scoring in dense windows - #9033

Open
BubbleCal wants to merge 1 commit into
yang/fts-stack-prepared-scoringfrom
yang/fts-stack-certified-and
Open

perf(fts): bound classic conjunction scoring in dense windows#9033
BubbleCal wants to merge 1 commit into
yang/fts-stack-prepared-scoringfrom
yang/fts-stack-certified-and

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Classic 4/5-term AND evaluation can decode and score every follower even when a partial exact score already proves the candidate cannot compete. Applying score-first evaluation universally is costly when posting overlap is low. This PR enables it only inside a metadata-certified dense range.

  • Prove continuous coverage from each current posting block's first/last/count metadata, while retaining actual follower membership checks.
  • Build conservative suffix bounds only for eligible live windows and lazily cache exact BM25 norm addends.
  • Preserve canonical query-order f32 scores and inclusive tie behavior.
  • Reuse an unbounded zero-floor window and invalidate it when local, shared, or compound floors become positive; clear eligibility before phrase evaluation.

The classic path remains used by compound WandCursor and explicit classic searches after #9030 enables standalone wide-query bulk routing.

Most of this diff is regression coverage: approximately 327 production-section additions and 1,334 test additions. Shared fixtures retain the distinct dense/sparse, malformed-metadata, short-tail, high-DocID, score-bit, tie, norm-fallback, seek/shallow, and floor-transition cases. No isolated main A/B result is claimed for this layer.

Validation

  • Per-layer inverted-index suite: 603 passed.
  • Required all-workspace Clippy and formatting: passed.
  • PR metadata/gatekeeper checks: pending. Full hosted test workflows only run for main/release/** bases; this intermediate layer has been tested and linted locally. The full hosted matrix will run after its base is retargeted to main following the parent merge.

Stack, 3/4: #9031#9032this PR#9030. Base: #9032. Uses the lazy posting-cursor support introduced in #9031.

Merge in stack order; retarget/restack the remaining PRs after their parent merges.

Certify dense posting ranges before progressive scoring, retain conservative
suffix bounds and exact query-order scores, and reuse zero-floor windows
and lazy norm lookup. Preserve sparse, phrase, and compound-cursor behavior.

Share test fixtures without removing assertions or parameter coverage.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The score-first path preserves canonical score bits and floor semantics in the focused regression set, and its strict eligibility keeps unsupported scorers and formats on the existing path.

The remaining risk is performance rather than correctness: this layer adds dense-window certification and a lazy norm cache without an isolated latency/CPU comparison, so its net benefit—and possible short-tail overhead—remains unverified. Keep the scope narrow until representative classic 4/5-term workloads confirm the trade-off; if the incremental gain is small, progressive scoring after full DocID alignment is the simpler alternative.

This PR is stacked on #9032, which depends on #9031; the safe landing order remains #9031#9032 → this PR.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant