Skip to content

Do not drop strictly competitive boundary candidates in filtered HNSW search#16409

Open
Incheonkirin wants to merge 2 commits into
apache:mainfrom
Incheonkirin:filtered-hnsw-boundary-parity
Open

Do not drop strictly competitive boundary candidates in filtered HNSW search#16409
Incheonkirin wants to merge 2 commits into
apache:mainfrom
Incheonkirin:filtered-hnsw-boundary-parity

Conversation

@Incheonkirin

Copy link
Copy Markdown
Contributor

Description

FilteredHnswGraphSearcher can stop before exploring a candidate whose score is equal to the current result floor, and it rejects neighbors whose score is exactly Math.nextUp(results.minCompetitiveSimilarity()).

The latter score is strictly better than the current minimum and is competitive. The base HnswGraphSearcher already handles both boundaries:

The filtered searcher was added before those follow-ups and retained the earlier boundary behavior. This change applies the same contract to the filtered path:

  • explore one candidate equivalent to the current minimum;
  • admit a neighbor at minAcceptedSimilarity;
  • gate the scored batch on maxScore > results.minCompetitiveSimilarity(), the same bound the base searcher uses;
  • reset the one-equivalent guard when collection raises the minimum.

Regression tests

Both tests call FilteredHnswGraphSearcher directly, so they do not depend on the filtered-search dispatch heuristic.

The first test uses a four-node graph with two accepted nodes and k=1. The entry node scores 0.5; its accepted neighbor scores Math.nextUp(0.5). Before this change, filtered HNSW returns the 0.5 entry and never considers the strictly better neighbor. After the change, it returns the neighbor.

The second test searches a chain where every node has the same score and asserts that visitation stays bounded: one candidate tied with the current minimum may be explored, then the search stops.

Validation

  • regression tests: fail on current main, pass with the change
  • ./gradlew :lucene:core:check passes: 8,534 tests, 307 skipped

Related history: #14160, #14215, #14249, #14366, #15500.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant