Searcher automatically skips specific queries #1714
-
|
When using BM25 searcher on MSMARCO, I found it automatically skips (retrieves no document) some special queries:
How to deal with this? (Currently I manually modified these queries so that they can be processed) |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
hi @namespace-Pt - The tokenization is determined by Lucene analyzer used during indexing (which has to match the one used during retrieval). You'll need to modify the analyzer if you want to deal with special cases. In general, in any large, real-world dataset, there are going to be queries/documents that are mangled/mis-tokenized/etc. It's usually not worth the effort to try to fix everything... |
Beta Was this translation helpful? Give feedback.
-
|
Thank you lintool. |
Beta Was this translation helpful? Give feedback.
hi @namespace-Pt - The tokenization is determined by Lucene analyzer used during indexing (which has to match the one used during retrieval). You'll need to modify the analyzer if you want to deal with special cases.
In general, in any large, real-world dataset, there are going to be queries/documents that are mangled/mis-tokenized/etc. It's usually not worth the effort to try to fix everything...