Ranking subset of the documents for given query

[rajharshiitb]

Hi,
First of all thanks for this wonderful library.
My question is, is there a way to find the score for a subset of the document for the given query?
For eg: Let's say index is generated for 100M docs but at the time of scoring, I may be interested in scores of 100k docs which is subset of 100M. Is there any method to do so?

Thanks and Regards,
Harsh

[lintool]

Hi @rajharshiitb - this isn't easy to do with Pyserini or even Lucene for that matter. What's your use case for this feature?

[rajharshiitb]

Hi @lintool, thanks for the reply. I am working on a project that has search space of 176M documents and first level of retrieval is BM25 based system (lucene for example). But, while retrieving the hits my search space is not same as 176M - now reduced. For eg. I want to only rank and retrieve only those docs which contains a certain entity (which i have knowledge of through a built entity_to_doc inverted index).
I have built an alternate solution by building lucene index for each entity (1271 entities: entities which has more than 20k docs indexed to it). It works fine but it can be said it lacks correct global word distribution.

Let me know if I can contribute something towards it (if you find it useful in general case and want to implement it)

Thanks,
Harsh

[lintool]

The "best practices" solution to your use case is to index the entities in a separate field, and then search across both the free-text field and the entity field. The entity field can have a "hard" (exact) match requirement. This can be done in Lucene, but anserini/pyserini have not exposed the proper hooks to make this easy in Python.