Skip to content
This repository was archived by the owner on Apr 29, 2026. It is now read-only.

rely on provider for counting of vectors and only optionally fallback on DB due to slow query#58

Merged
timothycarambat merged 1 commit into
masterfrom
rely-on-provider-for-counts
Sep 25, 2023
Merged

rely on provider for counting of vectors and only optionally fallback on DB due to slow query#58
timothycarambat merged 1 commit into
masterfrom
rely-on-provider-for-counts

Conversation

@timothycarambat

Copy link
Copy Markdown
Member

Until the data migration is done and organiztion_id is appended to document vectors we will need to rely on the provider for counting vectors.

  1. This can count documents or vectors that VectorAdmin isn't aware of because we are reading from remote
  2. The SQL query currently used is not great and with 50K documents results in an even larger ...IN(1,2,3,) query looking for document ids that are in document_vectors and would be easier to have a fixed organization_id key we can COUNT against since we will very easily reach the upper end of IN() parameters

@timothycarambat timothycarambat merged commit 03667ae into master Sep 25, 2023
@timothycarambat timothycarambat deleted the rely-on-provider-for-counts branch September 25, 2023 22:59
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant