Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add upsert flow in architecture doc #627

Merged
merged 6 commits into from
Aug 20, 2020
Merged

Add upsert flow in architecture doc #627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6fffbaa
add upsert flow
kevindiu Aug 13, 2020
372e667
fix CI
kevindiu Aug 13, 2020
28b2d66
apply suggestions
kevindiu Aug 14, 2020
b67af19
apply suggestion
kevindiu Aug 17, 2020
7386b2a
fix
kevindiu Aug 17, 2020
80c00a7
fix
kevindiu Aug 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions assets/docs/upsert_flow.drawio
View file
Open in desktop

Large diffs are not rendered by default.

Binary file added assets/docs/upsert_flow.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
16 changes: 15 additions & 1 deletion docs/overview/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,21 @@ When the user updates a vector from Vald:
24. Vald Meta Gateway will return success to the Vald Filter Gateway.
25. Vald Filter Gateway will return success to the Vald Ingress.

<!-- ### Upsert -->
### Upsert

<img src="../../assets/docs/upsert_flow.png" />
kevindiu marked this conversation as resolved.
Show resolved Hide resolved
kevindiu marked this conversation as resolved.
Show resolved Hide resolved

Upsert request updates the existing vector if the same vector ID exists, or inserts the vector into Vald.
When the user upsert a vector to Vald:

1. Vald Ingress receives the request from the user. The request includes the vector ID(s) and the vector(s).
2. Vald Ingress will forward the request to the Vald Filter Gateway to pre-process the request data.
3. Vald Filter Gateway will forward the request to the user-defined Vald Ingress Filter. After the Vald Ingress Filter received the request, it will perform the pre-processing logic defined by the user, for example, padding the vector to match the vector dimension in Vald.
4. After the request is processed by the user-defined Vald Ingress Filter, the result will return to the Vald Filter Gateway.
5. Vald Filter Gateway will forward the processed data to the Vald Meta Gateway. Vald Meta Gateway is used to check if the same vector with the same vector ID exists in Vald cluster.
6. Vald Meta Gateway will forward the request to the Vald Meta to check if the metadata, which contains the vector ID(s), exists or not.
7. Vald Meta gets the UUID(s) by vector ID(s). It returns an error if no UUID(s) is found.
8. If the vector with the same vector ID exists, Vald Meta Gateway will process the Update flow from step 5 to step 25. If the vector does not exist, Vald Meta Gateway will process the insert flow from step 5 to step 18.

### Delete

Expand Down