Skip to content

fix(fts): sum all boolean must scores - #8292

Merged
BubbleCal merged 1 commit into
mainfrom
yang/oss-1696-sum-boolean-must-scores
Aug 5, 2026
Merged

fix(fts): sum all boolean must scores#8292
BubbleCal merged 1 commit into
mainfrom
yang/oss-1696-sum-boolean-must-scores

Conversation

@BubbleCal

Copy link
Copy Markdown
Contributor

What is the bug?

Boolean FTS queries over committed indices only used the first MUST clause as a scoring clause. Later MUST clauses filtered membership but did not contribute to _score.

This made boosts on later clauses ineffective, allowed clause order to change rankings, and disagreed with the active MemWAL path, which already sums every MUST score.

What issues does this cause?

  • Same-column composable execution could return the wrong top-k order.
  • Cross-column fallback execution exposed the same first-score-only behavior.
  • Committed and active data could rank the same Boolean query differently.

How does this PR fix the problem?

  • Sum scores and conservative score bounds from every required child in the composable conjunction scorer.
  • Avoid propagating a parent competitive-score floor to one child when multiple required scores can combine to beat it.
  • Sum every _score column produced by the committed-index Boolean fallback.
  • Document MUST as a required scoring clause across Rust, Python, Java, and the FTS guide.
  • Add regression coverage for boosted clauses, reversed clause order, nested three-MUST queries, real top-k truncation, same-column composable execution, cross-column fallback execution, and active MemWAL scoring.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings
  • cargo test -p lance test_boolean_must_only
  • cargo test -p lance test_boolean_must_scores_sum_across_execution_paths
  • cargo test -p lance-index boolean_sums_all_matching_clause_scores
  • cargo test -p lance-index required_conjunction_uses_all_must_scores_for_competitive_bounds
  • cd python && uv run make lint

cd java && ./mvnw spotless:check could not be run because this environment has no JDK/JRE installed.

Scope

This is a correctness prerequisite for OSS-1696, not the complete ticket. It does not implement the specialized WAND/MAXSCORE, ReqOpt, or Phrase strategies, instrumentation, or the 10M-row MMLB benchmark, and it should not close OSS-1696.

Base-plus-active structured Boolean execution also remains unsupported by the MemWAL planner and is not expanded in this focused fix. There are no index-format or public API changes, and this PR makes no performance claim.

@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-docs Documentation bug Something isn't working labels Aug 5, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

Summing all required scores at the conjunction and fallback boundaries restores order-independent ranking and matches the existing MemWAL contract. Keeping multi-child competitive floors at the combined node preserves correctness while retaining conservative block pruning.

@BubbleCal
BubbleCal merged commit d7e9417 into main Aug 5, 2026
57 of 58 checks passed
@BubbleCal
BubbleCal deleted the yang/oss-1696-sum-boolean-must-scores branch August 5, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-docs Documentation A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-python Python bindings bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants