Loading passage binary codes from BinaryHash Faiss Index stored on disk

[thakur-nandan]

Hi @ikuyamada,

Thanks for the awesome repository and clean code for BPR!

When we have stored the passage embeddings (binary codes) as a faiss BinaryHash Index and saved it to the disk. Can we get the passage embeddings back given only loading back the faiss BinaryHash index? Or would we need to also separately save the passage embeddings (This would take more memory)?

Kind Regards,
Nandan Thakur

[ikuyamada]

Hi @NThakur20,

As far as I know, the raw passage embeddings cannot be reconstructed from the Binary Hash index in Faiss. If you use this type of index, you need to store the index separately in memory or a fast storage device with caching such as memory mapped file.

[thakur-nandan]

HI @ikuyamada, Thanks for your prompt reply.

Indeed this was not possible with BinaryHash Index, went ahead with BinaryFlat instead which allows reconstruction!