-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Precision issue while querying the vector on the Qdrant vector benchmark test #56
Comments
I had some buddy-related issues reproducing that behavior locally. If you have a pre-built index (preferably with one chunk), please post a link. |
@glookka pls find it here |
The fixed dump is here |
I optimized the index to a single chunk and ran queries with
|
Also CREATE TABLE looks like this:
It uses |
I identified and fixed the issue in 694cbe5 |
@glookka how severe was the issue? In what circumstances could it affect the search results relevancy? |
It affected any search with k>limit. So it was quite severe. |
While trying to integrate Manticore Search into the Qdrant vector database benchmark, I discovered that we have very low mean precision.
In the current test that I have attached here, Manticore Search's mean precision equals 0.16, while Qdrant's is 0.99.
We should investigate this behavior and try to understand why it happens.
I have also attached the original test set in the attachments to initialize the database.
I checked also it with https://newtum.com/calculators/maths/cosine-similarity-calculator
So Manticore top 1 result gives 0.89 similarity while the expected one is 0.91.
Path to the dump:
/tmp/bench.tar.gz
ondev2
.The original Request vector used for search [k=100, ef=128]
Expected result with doc IDS
Original response in format (id, knn_dist) by manticoresearch
Original response with score (looks like knn dist) in QDRant
doc id 81360 vector
doc id 786559 vector
The text was updated successfully, but these errors were encountered: