Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: HNSW index fails if initial insert has doc id > 24999 #1848

Closed
2 tasks done
etiennedi opened this issue Mar 9, 2022 · 0 comments · Fixed by #1851
Closed
2 tasks done

Bug: HNSW index fails if initial insert has doc id > 24999 #1848

etiennedi opened this issue Mar 9, 2022 · 0 comments · Fixed by #1851
Assignees
Labels

Comments

@etiennedi
Copy link
Member

etiennedi commented Mar 9, 2022

Problem

Previously it was impossible for a doc id of the initial insert into HNSW to be larger than its size. But since we added the ability to add objects without a vector and introduce the vector later it is now possible to import more than 25,000 objects before ever making use of the HNSW index. In that case, we now need to check if the initial size is large enough and adjust it accordingly.

Goals

  • The initial insert will work with any doc id regardless of initial index size
  • Add tests to prevent regression

Notes

While we can reproduce this with a stress test, I think we should also be able to reproduce this with a very simple integration test in the adapters/repos/db/vector/hnsw package. A simple insert with doc id 25,000 on an empty index should already reproduce this.

@etiennedi etiennedi added the bug label Mar 9, 2022
@antas-marcin antas-marcin self-assigned this Mar 9, 2022
etiennedi added a commit that referenced this issue Mar 10, 2022
…x-fails-if-initial-insert-has-doc-id-gt-24999

gh-1848 Bug: HNSW index fails if initial insert has doc id > 24999
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants