Skip to content

Correct Bayesian score fusion semantics and pruning#16410

Open
jaepil wants to merge 5 commits into
apache:mainfrom
jaepil:bayesian-bm25-fusion-corrections
Open

Correct Bayesian score fusion semantics and pruning#16410
jaepil wants to merge 5 commits into
apache:mainfrom
jaepil:bayesian-bm25-fusion-corrections

Conversation

@jaepil

@jaepil jaepil commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Description

This fixes probability semantics and competitive pruning for BayesianScoreQuery and LogOddsFusionQuery.

The previous fusion path applied softplus gating by default, which discarded negative log-odds evidence, and could propagate competitive-score pruning to individual signals. Pruning an individual negative signal changes matching negative evidence into a neutral non-match and can incorrectly increase the fused score. Single-active-signal leaves and MatchNoDocsQuery rewrites could also bypass or change the declared fusion model.

The change:

  • preserves signed log-odds internally while returning a non-negative sigmoid probability
  • keeps softplus gating as an explicit opt-in and retains existing constructor signatures
  • supports per-signal prior removal and a single target prior
  • preserves neutral signals across rewrites and single-active-signal leaves
  • applies outer boosts after Bayesian calibration and fusion
  • performs block-max skipping only from a safe upper bound on the final fused score
  • makes BayesianScoreEstimator use all matching live documents rather than a capped top-doc sample based on maxDoc

Validation

  • ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestBayesianScoreQuery --tests org.apache.lucene.search.TestLogOddsFusionQuery
  • ./gradlew :lucene:core:check -x checkLicenses (8,548 tests passed, 309 skipped)
  • ./gradlew checkLicenses
  • ./gradlew :lucene:core:tidy :lucene:core:javadoc
  • uv run --project dev-tools prek run --files <changed files>

@jaepil
jaepil marked this pull request as ready for review July 23, 2026 00:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant