Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data Transforms Schema Registry Rust SDK #16464

Merged
merged 8 commits into from
Mar 18, 2024
Merged

Data Transforms Schema Registry Rust SDK #16464

merged 8 commits into from
Mar 18, 2024

Conversation

rockwotj
Copy link
Contributor

@rockwotj rockwotj commented Feb 3, 2024 •
edited

Fixes: #16075

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x
  • v23.1.x

Release Notes

Features

  • Data Transform's Rust SDK now supports a Schema Registry Client.

@github-actions github-actions bot added area/build area/wasm WASM Data Transforms labels Feb 3, 2024
@rockwotj rockwotj mentioned this pull request Feb 3, 2024
Closed
@rockwotj rockwotj self-assigned this Feb 3, 2024
@rockwotj rockwotj marked this pull request as ready for review February 5, 2024 16:16
@rockwotj rockwotj requested a review from a team as a code owner February 5, 2024 16:16
@rockwotj rockwotj requested review from andrewhsu, michael-redpanda and a team and removed request for a team February 5, 2024 16:16
@rockwotj
Copy link
Contributor Author

rockwotj commented Feb 5, 2024 •
edited

@redpanda-data/documentation

As I mentioned last week. This PR introduces new API surface for the Rust SDK by adding a schema registry client.

The only files you need to review are:

Thanks in advance!

@kbatuigas kbatuigas self-requested a review February 9, 2024 18:46
@rockwotj
Copy link
Contributor Author

rockwotj commented Feb 9, 2024 •
edited

force push: Fix a couple typos

kbatuigas
kbatuigas reviewed Feb 10, 2024
View reviewed changes
Copy link

@kbatuigas kbatuigas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestions for some minor edits

src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr-types/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr/src/lib.rs Outdated Show resolved Hide resolved
src/transform-sdk/rust/sr/src/lib.rs Outdated Show resolved Hide resolved
@rockwotj rockwotj requested a review from kbatuigas March 4, 2024 15:33
kbatuigas
kbatuigas approved these changes Mar 8, 2024
View reviewed changes
Copy link

@kbatuigas kbatuigas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @rockwotj !

@rockwotj
transform-sdk/rust/publish: improve script
53c748f 
We're going to add a few more crates for schema registry, so improve the
publish script so it scales a little better.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust: split out varint crate
235a436 
We need some of these varint routines for the schema registry API, so
split those out into it's own crate.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/go/sr: update docs links
6e78b99 
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust/sr: add types crate
12e9b5b 
In a similar structure to the core crates, add a type crate to share
types between the different sr crates.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust: derive equality
73a06c6 
For types users could see derive equality for easier testing.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust/sr: add ABI
6507ad4 
Add a Rust version of our schema registry ABI, and wrap it in the schema
registry client trait.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust/sr: add client
dec2aa3 
Add public facing crate that exposes functionality to access Redpanda's
built-in Schema Registry.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust/sr: documentation updates
bf85aae 
Based on the review from our talented documentation team, update some of
the documentation we have available.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
Copy link
Contributor Author

Force push: rebase

michael-redpanda
michael-redpanda approved these changes Mar 18, 2024
View reviewed changes
Copy link
Contributor

@michael-redpanda michael-redpanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One possible area of expansion would to allow the user to fetch and set schema compatibility.

@rockwotj
Copy link
Contributor Author

One possible area of expansion would to allow the user to fetch and set schema compatibility.

Yes, I do wonder if this is a control plane or data plane action in SR. I've only exposed the minimal set of things I believe we need for "data plane" usage of SR

@rockwotj rockwotj merged commit b957ade into redpanda-data:dev Mar 18, 2024
18 checks passed
@rockwotj rockwotj deleted the sr-rs branch March 18, 2024 15:49
@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-16464-v23.3.x-443 remotes/upstream/v23.3.x
git cherry-pick -x 53c748f99e5534aee3e7abea596aace429139032 235a436d348f2df8388d9b4285a0b9b4f0f4d5e6 6e78b99c0d49b7e7313a916e68c73f46991b7e44 12e9b5b9594a4ec505981318fa1a70b613234bbf 73a06c6e25f0eef2851831b52d787f3b8fe416fd 6507ad40a0e0455de8ba43bd8317c2e56a62ecd7 dec2aa32ddb1fe4c282888a3db7f49e5b905123e bf85aae0ea65cd75f223ef0dc8004f92cca20606

Workflow run logs.

@vbotbuildovich vbotbuildovich mentioned this pull request Mar 18, 2024
Closed
@rockwotj rockwotj mentioned this pull request Mar 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kbatuigas kbatuigas kbatuigas approved these changes

@michael-redpanda michael-redpanda michael-redpanda approved these changes

@andrewhsu andrewhsu Awaiting requested review from andrewhsu andrewhsu is a code owner automatically assigned from redpanda-data/devprod

@voutilad voutilad Awaiting requested review from voutilad

Assignees

@rockwotj rockwotj

Labels
area/build area/wasm WASM Data Transforms
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Provide Schema Registry support in the Rust Data Transforms SDK
5 participants
@rockwotj @vbotbuildovich @voutilad @kbatuigas @michael-redpanda