what does parameter "PQ_disk_bytes" mean?

[HannanKan]

I will appreciate if you can tell me what "PQ_disk_bytes" parameters controls. Some hint by source code are "(for very large dimensionality, use 0 for full vectors)". I still can not get it.

[ShikharJ]

@harsha-simhadri

[ShikharJ]

@rakri Would you be able to provide some insight?

[rakri]

For very large dimensional datasets, say, > 1024, as implemented DiskANN will crash as the contents of a single node exceed the 4KB disk sector. In such situations (or when you want lower disk footprint), we can store a separate compressed representation on the disk index file. The number of bytes per vector you want as part of the disk footprint is that parameter. If you specify 0, it will store the vectors in full precision form occupying dim*sizeof(data-type) bytes per point

[HannanKan]

Got it. Thank you!