New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix reconstruction error discussed in #15 #16
Conversation
@@ -292,7 +292,8 @@ def index_embeddings( | |||
metric_type = "l2", | |||
max_index_memory_usage = max_index_memory_usage, | |||
current_memory_available = current_memory_available, | |||
should_be_memory_mappable = True, | |||
make_direct_map = True | |||
should_be_memory_mappable = False, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh you can actually keep the memory mapping, it will apply to everything except for the direct map
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was getting the same error when I did this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you send the full stack trace ? I don't understand
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @rom1504 here is the full stack trace:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
[<ipython-input-3-7b2653026171>](https://localhost:8080/#) in <module>()
32 knn_extra_neighbors = 100, # num extra neighbors to fetch
33 max_index_memory_usage = '100m',
---> 34 current_memory_available = '1G'
35 )
36
11 frames
[/usr/local/lib/python3.7/dist-packages/faiss/swigfaiss.py](https://localhost:8080/#) in reconstruct_n(self, i0, ni, recons)
1997 :param recons: reconstucted vector (size ni * d)
1998 """
-> 1999 return _swigfaiss.Index_reconstruct_n(self, i0, ni, recons)
2000
2001 def search_and_reconstruct(self, n, x, k, distances, labels, recons):
RuntimeError: Error in virtual void faiss::Index::reconstruct(faiss::Index::idx_t, float*) const at /project/faiss/faiss/Index.cpp:48: reconstruct not implemented for this type of index
You can use this colab to reproduce: https://colab.research.google.com/drive/1nvAxBGHnjrNFmgBm8Qu9ZRnmvwQyCFLI?usp=sharing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok i found the problem. You installed faiss-gpu (first line of the colab). Don't do that, use faiss-cpu instead.
Then you can revert this PR
This PR fixes the issue with reconstruction of the faiss index. One caveat is that we can no longer do memmapping to reduce RAM overhead. Maybe this will be fixed in faiss soon, but for now memory will be an issue for extremely large indices.