Querying on Subset of Document_IDs

[PrimoUomo89]

Would love to be able to pass in an array of document_IDs as argument to query function, representing the subset of documents to query. Not familiar enough with the inner workings of the technology to propose a resolution myself. Would gladly take some guidance from someone more senior so I can produce a pull request myself.

[bclavie]

Hey, this is a great suggestion and has already been requested!

It's doable without too many code changes because ColBERT natively supports querying only a subset of passage IDs. @anirudhdharmarajan can talk a bit more about this but right now we do have a mapping of document_ids to internal pids (passage_ids) at indexing time. This mapping could be used to make it so search() would only look within pids that are mapped to document_ids specified by the user, producing the behaviour you're requesting.

[PrimoUomo89]

I had looked a bit at the pids when reviewing @anirudhdharmarajan 's pull request for document_metadata so a little familiar with what you're describing. I'll take a rough swing this week, but it's possible I'm out of my depth. Will look for feedback when I have a a plan.

[PrimoUomo89]

Deleted my previous comments because it had the wrong approach. Will send a pull request later today hopefully.

[PrimoUomo89]

Pull Request complete.

[hehuan2363]

Are you refering to something similar to this issue in Colbert: stanford-futuredata/ColBERT#304

[PrimoUomo89]

@hehuan2363
The main difference between what I implemented for this issue and what they're discussing on that issue, is that instead of implementing access to the filter lamda function, I instead implemented access to submitting document_ids (which are resolved to pids before search). I'm not sure what the best implementation would be on this side, but I imagine a lamda function which resolves document_ids in RAGatouille, which would then resolve the pids to search. If someone asks for it, I'll help out with that.

[bclavie]

Pull

Thank you! Closing this issue as it's been merged