Add a rescorer that uses DoubleValuesSource values to re-score first pass hits #14776
Conversation
github-project-automation
bot
moved this to Open
in OpenSearch Lucene & Core Performance Tracking
|
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR. |
| @Override | ||
| public Explanation explain(IndexSearcher searcher, Explanation firstPassExplanation, int docID) | ||
| throws IOException { | ||
| Explanation first = |
There was a problem hiding this comment.
Maybe it should be moved to when it is needed, such as L120?
There was a problem hiding this comment.
My flow was to produce first pass explanation, then second explanation, then combined score and final explanation.
| } | ||
| } | ||
| if (leafWithDoc == null) { | ||
| throw new IllegalArgumentException( |
There was a problem hiding this comment.
Why we are throwing exception here instead of Explanation.noMatch?
There was a problem hiding this comment.
Yeah, this is a tricky one. Since this explain API can take any docId, I can see the argument of surfacing a "noMatch" here. However, my thinking is that Rescorer#explain API is meant to explain how the Rescorer would score this docId for given searcher, when rescore() is called.
If we hit null here, it means that the docId was not matched by the provided searcher in the first-pass itself. This is likely because a different searcher was passed than the one that produced provided docId, which is an error that we should surface.
Suppose we say noMatch for first pass, and the docId does have a value in DoubleValuesSource. Then the explain API will return a score for the docId, However, the actual rescore API will never return a score for this (searcher, docId) combination because they don't match.
There was a problem hiding this comment.
Gotcha, thanks for the explanation. Maybe we can add some nuance you mentioned above to the error message, if the only possible case is a different IndexSearcher was passed? Anyhow, no strong opinion here.
vigyasharma
force-pushed
the
rescorer_dbv
branch
from
b65f6de to
0a42fee
Compare
github-project-automation
bot
moved this from Open
to Merged
in OpenSearch Lucene & Core Performance Tracking
A
Rescorerthat uses values from providedDoubleValuesSourceto re-score top N hits of a query.Enables us to rescore hits from ANN vector queries using full-precision or late-interaction DoubleValuesSources (#14729 and #14708).