-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add Vector Search indexers and retrievers to Vertex AI plugin #519
base: main
Are you sure you want to change the base?
Conversation
ab71154
to
c33639e
Compare
45b48da
to
8e3b0aa
Compare
8e3b0aa
to
be7cc48
Compare
f7a43ea
to
372313b
Compare
4d489b2
to
51c0c8f
Compare
51c0c8f
to
f7ed986
Compare
let retrievers: RetrieverAction<z.ZodTypeAny>[] = []; | ||
|
||
if (options?.vectorSearchIndexOptions) { | ||
const defaultEmbedder = embedders[0]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we also let the user specify an embedder (rather than always using a default one)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should be, though i could have made this more obvious -
options?: Options | ||
) => Promise<string[]>; | ||
|
||
export interface VectorSearchIndexOption< |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional: Have you considered naming this VectorSearchOptions
? The inclusion of Index
might suggest that this is only relevant for indexing and not retrieving. Although Index here means a DB index I suppose (rather than the act of indexing).
WDYT?
cc: @pavelgj
js/plugins/vertexai/tests/vector-search/query_public_endpoint_test.ts
Outdated
Show resolved
Hide resolved
js/plugins/vertexai/tests/vector-search/query_public_endpoint_test.ts
Outdated
Show resolved
Hide resolved
js/plugins/vertexai/tests/vector-search/upsert_datapoints_test.ts
Outdated
Show resolved
Hide resolved
Thanks @cabljac for adding out-of-the-box indexers/retrievers and test coverage :) I left a few comments on you latest updates. |
664b595
to
3023575
Compare
3023575
to
362673e
Compare
This PR adds Vertex AI Vector Store to the Vertex AI plugin. There are some design decisions still to be made, as VVS works differently to other vector databases (in that it doesn't store metadata, only ids and the feature vectors)
I have included a samples for BigQuery, Firestore, and a Custom local document store (just in a JSON file)