New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x8 vs x4fsr #124
Comments
Hello! Autofaiss will do a binary search to find the best set of hyperparameters and will set the lower exploration bound given the targeted minimum number of neighbors to retrieve (20 according to the logs). In your case, it seems that the function estimating the output coverage returned 10.65% for the higher bound of the exploration window (see code: get_nearest_neighbors_coverage). I see several possible explanations:
In order to help you more, I would need more context on the commands you used to get these results PS: Autofaiss doesn't support the |
Thank you for your kind help and this extensive answer. My index is not empty, my nvectors is correct. The thing which is weird is that I only change the encoding in my script by putting x8 instead of x4fsr. I will continue to investigate on my side. Yeah, I know that you do not support it. I forked your repo to build it, in order to modify build_index to be able to index any valid index_key even if it means that there is no hyperparameter tuning |
What can we do to prevent this?
This happened with "OPQ768_768,IVF262144_HNSW32,PQ768x8" -> bad max coverage
With the index_key "OPQ768_768,IVF262144_HNSW32,PQ768x4fsr", everything was ok. The vectors were just a bit too compressed.
My d is 768.
Thank you
The text was updated successfully, but these errors were encountered: