You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pg_sparse is an extension that enables similarity search over sparse vectors in Postgres with HNSW. Think of it as pgvector for sparse vectors.
Why
pgvector only supports dense vectors up to 2K dimensions; sparse vectors are much higher dimensionality and require a custom storage + HNSW implementation.
Todo LIst
Option parsing (m, ef, etc.)
Import hnswlib rust bindings as crate
Create HNSW index on disk
Insert sparse vectors into HNSW index
Perform index scan
Delete vectors
Implement vacuum
Implement correct cost estimate function
Index resizing
Pass in ef_search
Tests
Wrote basic unit and regression tests. For testing instructions see the extension README
@rebasedming sorry to keep making requests, but could you add the workflows to publish pg_sparse and to test pg_sparse? You can mimick them from our pg_bm25 workflows. I think once you do, you'll notice you need to add pg=16 in the places in code where you have pg=15 feature flags to make it supported, as it's going to fail the tests on pg16 otherwise (it's not enough to just add it to the Cargo.toml)
@rebasedming sorry to keep making requests, but could you add the workflows to publish pg_sparse and to test pg_sparse? You can mimick them from our pg_bm25 workflows. I think once you do, you'll notice you need to add pg=16 in the places in code where you have pg=15 feature flags to make it supported, as it's going to fail the tests on pg16 otherwise
The reason will be displayed to describe this comment to others. Learn more.
high-level lgtm! Thank you for adding the test workflow. It's missing the deployment workflow, but I can handle that in a separate PR, this one has already grown so big
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rebasedming
force-pushed
the
feat/pg_sparse
branch
from
Ticket(s) Closed
What
pg_sparseis an extension that enables similarity search over sparse vectors in Postgres with HNSW. Think of it aspgvectorfor sparse vectors.Why
pgvectoronly supports dense vectors up to 2K dimensions; sparse vectors are much higher dimensionality and require a custom storage + HNSW implementation.Todo LIst
m,ef, etc.)hnswlibrust bindings as crateef_searchTests
Wrote basic unit and regression tests. For testing instructions see the extension README