Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support two indexing styles: ColBERT/PLAID style optimisation and HNSW-style uncompressed indexes #110

Open
bclavie opened this issue Feb 4, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request help wanted Extra attention is needed Indexing expansion

Comments

@bclavie
Copy link
Owner

bclavie commented Feb 4, 2024

Currently, we only use the ColBERT optimised indexes, or index-free in-memory encodings.

For low-to-medium volumes of documents, not using the ColBERT optimisation can have advantages: easier CRUD, potentially better performance, the concept is easier to grasp, quicker indexing time... (and tradeoffs: much bigger footprint, no constant-time querying, etc...)

We should provide the ability to create both kinds of indexes, and let the user decide which indexing style they wish to use.

@bclavie bclavie added enhancement New feature or request help wanted Extra attention is needed Indexing expansion labels Feb 4, 2024
@bclavie bclavie self-assigned this Feb 4, 2024
@0-hero
Copy link

0-hero commented Feb 13, 2024

Are you looking to use something like FAISS or https://github.com/nmslib/hnswlib ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed Indexing expansion
Projects
None yet
Development

No branches or pull requests

2 participants