Pointers on using this for retrieval-reader problem with growing context

[lawliet19189]

Thanks for the great work! I have read the paper and understood the Gist of the approach but I am out-of-depth on few things so I wanted to clarify few things here and get some suggestion.
My problem statement is straight-forward, I have thousands of documents as context and I need to retrieve an answer for query in real-time. I came across your approach and I wanted to try it out and possibly adapt this in industry setting. One other constrain is that the context grows, that is, new documents get added on and the retrieval needs to support that.

Do you think your approach is suitable for this use-case?

Thanks in Advance

[jhyuklee]

Hi @lawliet19189, thank you for your interest in our work.

Yes. you can use your new documents to create your own phrase index. You can see here for creating a phrase index.

To add new documents to the existing phrase index. you can additionally run our model to get phrase representations for the new documents, and add it to the existing faiss index. This script creates a phrase index from scratch with faiss, and you can use this script to add your new phrase representations as well (may require some understanding on the code + modification of the code, too).

Thanks.