Index does not fit to RAM

[littlewine]

Hi,
I am trying to run inference (retrieve.py)in a quite big collection and I get the following error :

RuntimeError: [enforce fail at CPUAllocator.cpp:64]. DefaultCPUAllocator: can't allocate memory:you tried to allocate 664833940224 bytes. Error code 12 (Cannot allocate memory) 

To my understanding, that means that colbert requires 600+ gb of memory, which the machine does not have. Is there any way to bypass this issue?

Can I read the file from an SSD and how much slower would the process be?

Some colleagues also suggested using huggingface.co datasets for that but I'm unsure whether this is feasible or where to start!

[okhat]

Might want to use batch retrieval and batch re-ranking? Brief instructions are in the README.

It does search in two steps, but doesn't have to load the full index into memory.

[okhat]

Alternatively, you might be interested in our binarization branch. It makes the index much smaller. It's still a work in progress; we'll improve its quality in the next 2-3 weeks. (Until then, you might lose a couple of points of quality if you use the binarization branch.)

[okhat]

@littlewine Is this resolved?

[littlewine]

Yes indeed! Thanks for your help!

[Lim-Sung-Jun]

Might want to use batch retrieval and batch re-ranking? Brief instructions are in the README.

It does search in two steps, but doesn't have to load the full index into memory.

Can you explain why you didn't add 512 in batch_retrieval?