Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ext/kv): connect to remote database #20178

Merged
merged 14 commits into from
Aug 22, 2023
Merged

Conversation

losfair
Copy link
Member

@losfair losfair commented Aug 16, 2023

This patch adds a remote backend for ext/kv. This supports connection to Deno Deploy and potentially other services compatible with the KV Connect protocol.

@losfair losfair requested a review from lucacasonato August 16, 2023 14:14
ext/kv/remote.rs Outdated Show resolved Hide resolved
Copy link
Member

@bartlomieju bartlomieju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any tests that could be added?

.github/workflows/ci.yml Show resolved Hide resolved
Comment on lines +10 to +11
- SQLite - backed by a local SQLite database. This backend is suitable for
development and is the default when running locally.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI people have been asking about alternative local backends: #18975 can this be addressed somehow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note on additional backends

ext/kv/README.md Outdated Show resolved Hide resolved
ext/kv/README.md Outdated Show resolved Hide resolved
Copy link
Member

@ry ry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add instructions on how to install protobuf for linux, windows, and osx as this will be a new requirement for those running "cargo install" or building. These should be in:

@losfair losfair force-pushed the kv-backend-remote branch from b54c2b1 to 7793e7c Compare August 18, 2023 12:29
@losfair
Copy link
Member Author

losfair commented Aug 18, 2023

Are there any tests that could be added?

Added a test with a mock remote target in test_util.

@losfair
Copy link
Member Author

losfair commented Aug 18, 2023

@losfair losfair requested review from bartlomieju and ry August 18, 2023 13:08
@@ -72,6 +80,9 @@ const PORT: u16 = 4545;
const TEST_AUTH_TOKEN: &str = "abcdef123456789";
const TEST_BASIC_AUTH_USERNAME: &str = "testuser123";
const TEST_BASIC_AUTH_PASSWORD: &str = "testpassabc";
const KV_DATABASE_ID: &str = "55bd5c48-58c2-4eb4-8adb-995c0763622a";
const KV_ACCESS_TOKEN: &str = "578528e20364923229b08f94386b236f";
const KV_DATABASE_TOKEN: &str = "iFOw5tcudKE2Cy4MKNuWZjxGkg4";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume these are just mock values not real tokens? Can you make them more obviously such - for example:

const KV_DATABASE_ID: &str = "11111111-1111-1111-1111-111111111111";
const KV_ACCESS_TOKEN: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
const KV_DATABASE_TOKEN: &str = "MOCKMOCKMOCKMOCKMOCKMOCKMOCK";

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test where the first endpoint returns the wrong JSON, and one where it returns the right structure but with a different protocol version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@losfair losfair requested a review from ry August 18, 2023 15:26
Comment on lines +1957 to +1959
const db = await Deno.openKv(
"http://localhost:4545/kv_remote_authorize_invalid_format",
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we should try to get the metadata during Deno.openKv and throw already in the open call if we can't authenticate / the metadata response is wrong.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That slows down app startup - any user initialization logic after Deno.openKv would have to wait for the first KV metadata response.

I prefer to keep that an asynchronous operation and defer any error to the first KV API call.

Copy link
Member

@ry ry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

there is some remaining concerns about introducing protoc as a build dependency - so we might need to come back and check-in the generated code.

@losfair losfair merged commit 6d4a005 into denoland:main Aug 22, 2023
@losfair losfair deleted the kv-backend-remote branch August 22, 2023 05:56
@kitsonk
Copy link
Contributor

kitsonk commented Aug 25, 2023

Just want to comment that this threw me for a loop when I pulled main and did a compile. I suspect a lot of folks used to building from source will scratch their head as the cargo build error don't give any hints as to how to correct it and I actually had to read through the PR to figure out what to do. Just has been years that cargo build was all you needed to do.

On the other hand, this is an awesome cool feature! 👍

@eternalphane
Copy link

Just want to comment that this threw me for a loop when I pulled main and did a compile. I suspect a lot of folks used to building from source will scratch their head as the cargo build error don't give any hints as to how to correct it and I actually had to read through the PR to figure out what to do.

Maybe we should add a feature to switch the remote backend?

bartlomieju pushed a commit that referenced this pull request Aug 31, 2023
This patch adds a `remote` backend for `ext/kv`. This supports
connection to Deno Deploy and potentially other services compatible with
the KV Connect protocol.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants