transform-sdk: experimental C++ support #16229

rockwotj · 2024-01-22T18:55:14Z

Add experimental C++ SDK. It's currently not going to be more formally supported in docs or RPK at the moment.

Most of the implementation here falls out from mirroring similar patterns from the Rust SDK.

Backports Required

Release Notes

none

vbotbuildovich · 2024-01-22T23:49:53Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44114#018d3347-1914-4f27-9b23-ed60b7a7f080

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44410#018d4fa3-2a33-4e0c-aaad-bec02b6ed202

andrewhsu

LGTM w.r.t. github workflow

rockwotj · 2024-01-28T09:19:38Z

/ci-repeat

oleiman

Looks good. As requested focused mostly on the headers and docs. If you're not opposed to squashing up some of the implementation, happy to go through the .cc in more detail as well.

oleiman · 2024-03-13T05:12:26Z

src/transform-sdk/cpp/README.md

+# Experimental Redpanda Data Transforms C++ SDK
+
+This directory contains an experimental C++ SDK for Redpanda Data Transforms.
+C++23 is required for both language features and standard library support.


oleiman · 2024-03-13T05:15:43Z

src/transform-sdk/cpp/README.md

+When writing C++ in WebAssembly, note that certainly language features are not supported.
+Noteably C++ exceptions, `setjmp`/`longjmp`, threads and networking is not supported.


q: are some of these fundamental limitations of wasm as a c++ target, or just more things we don't currently support in our wasi implementation?

This is more of a wasm32 VM feature we're missing. There is no VM support for non local control flow. It's likely going to happen this year, until then we're stuck with -fno-exceptions (honestly I kind of like it tho).

Cool. I was wondering whether it's worth segmenting into like

things we don't support

things the VM doesn't support

things that can't or will never exist

But given that we're not offering an exhaustive list, maybe not worth it.

Basically all of these are things are not supported by the VM at the moment (except networking is a wasi limitation). We hope to lift them all in the future (except probably threads). I will add some color here.

src/transform-sdk/cpp/transform_sdk.h

src/transform-sdk/cpp/transform_sdk.cc

src/transform-sdk/cpp/README.md

src/transform-sdk/cpp/transform_sdk.h

src/transform-sdk/cpp/src/transform_sdk.cc

dotnwat · 2024-03-14T02:00:27Z

src/transform-sdk/cpp/transform_sdk.cc

std::string generally signifies the data is a valid UTF-8 data,

not sure i understand this part. afaict the standard says std::string stores 1-byte chars it seems like a fine container for bytes. either way tho, uint8_t is better and std::byte also exists.

Yeah sorry, this std::string should be string (UTF-8) data is sort of an assumption that I have picked up from my time at Google, but I do think std::vector<uint8_t> communicates the semantics better. I looked at std::byte and it doesn't plug into existing libraries as well, so I picked something that should integrate a little better with other C/C++ code.

dotnwat · 2024-03-14T02:01:03Z

src/transform-sdk/cpp/examples/identity.cc

+    redpanda::on_record_written(
+      [](redpanda::write_event event, redpanda::record_writer* writer) {
+          return writer->write(event.record);
+      });


well... that's neat

Glad you like it :)

Hopefully this is a rather easy interface for folks to use. It's very similar to what Rust/Go looks like.

rockwotj · 2024-03-14T02:34:33Z

Force push: Address comments and cleanup the history

Add an experimental C++ SDK. This SDK closely follows the Rust implementation in terms of API. We intentionally keep the source to be a single implementation and header file for easy drop into existing or other build systems. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Add a simple CMake build for the C++ SDK. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Closely mirrors the config in the root of the Redpanda repo. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Add an example of using the C++ SDK for the identity transform. This inclues a CMake build to integrate the SDK. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj · 2024-03-14T02:34:48Z

Force push rebase against dev

Add a README telling folks how to use the SDK along with a short overview. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj · 2024-03-14T02:40:01Z

If you're not opposed to squashing up some of the implementation, happy to go through the .cc in more detail as well.

I squashed it up, still up to you if you want to go through the .cc more! Thanks again for the review

oleiman

Took a non-exhaustive look at transform_sdk.cc, and it looks pretty great. I don't want to hold this up any longer.

src/transform-sdk/cpp/src/transform_sdk.cc

github-actions bot added area/build area/wasm WASM Data Transforms labels Jan 22, 2024

rockwotj force-pushed the cpp-transform-sdk branch from ad8aa26 to ad44850 Compare January 22, 2024 20:35

rockwotj self-assigned this Jan 22, 2024

rockwotj marked this pull request as ready for review January 22, 2024 21:15

rockwotj requested a review from a team as a code owner January 22, 2024 21:15

rockwotj requested review from andrewhsu, BenPope, oleiman, graphcareful and michael-redpanda and removed request for a team January 22, 2024 21:15

andrewhsu previously approved these changes Jan 23, 2024

View reviewed changes

rockwotj dismissed andrewhsu’s stale review via db08867 February 19, 2024 10:01

rockwotj force-pushed the cpp-transform-sdk branch 4 times, most recently from ce4c935 to 93279f5 Compare February 19, 2024 10:22

rockwotj force-pushed the cpp-transform-sdk branch from 93279f5 to cb932f3 Compare March 1, 2024 20:34

rockwotj removed request for BenPope, oleiman, graphcareful and michael-redpanda March 1, 2024 20:34

rockwotj force-pushed the cpp-transform-sdk branch 5 times, most recently from f3d9517 to e388282 Compare March 4, 2024 17:18

rockwotj requested a review from oleiman March 11, 2024 15:34

rockwotj force-pushed the cpp-transform-sdk branch 3 times, most recently from 8ea48cb to 8e12c31 Compare March 11, 2024 18:49

oleiman reviewed Mar 14, 2024

View reviewed changes

dotnwat reviewed Mar 14, 2024

View reviewed changes

rockwotj force-pushed the cpp-transform-sdk branch from 8e12c31 to 17262a1 Compare March 14, 2024 02:34

rockwotj added 4 commits March 13, 2024 21:34

transform-sdk/cpp: Add CMake integration

656bd22

Add a simple CMake build for the C++ SDK. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

transform-sdk/cpp: add clang-tidy config

e55058b

Closely mirrors the config in the root of the Redpanda repo. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

transform-sdk/cpp: add example

5538aca

Add an example of using the C++ SDK for the identity transform. This inclues a CMake build to integrate the SDK. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj force-pushed the cpp-transform-sdk branch from 17262a1 to b831ca7 Compare March 14, 2024 02:34

rockwotj added 2 commits March 13, 2024 21:38

transform-sdk/cpp: add README

ccbef1d

Add a README telling folks how to use the SDK along with a short overview. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

gha/transform: add C++ SDK CI

33a9c12

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj force-pushed the cpp-transform-sdk branch from b831ca7 to 33a9c12 Compare March 14, 2024 02:38

oleiman self-requested a review March 14, 2024 15:05

oleiman approved these changes Mar 15, 2024

View reviewed changes

src/transform-sdk/cpp/src/transform_sdk.cc Show resolved Hide resolved

src/transform-sdk/cpp/src/transform_sdk.cc Show resolved Hide resolved

rockwotj merged commit d0ec79a into redpanda-data:dev Mar 15, 2024
18 checks passed

rockwotj deleted the cpp-transform-sdk branch March 15, 2024 18:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

transform-sdk: experimental C++ support #16229

transform-sdk: experimental C++ support #16229

rockwotj commented Jan 22, 2024 •

edited

vbotbuildovich commented Jan 22, 2024 •

edited

andrewhsu left a comment

rockwotj commented Jan 28, 2024

oleiman left a comment

oleiman Mar 13, 2024

oleiman Mar 13, 2024

rockwotj Mar 14, 2024

oleiman Mar 14, 2024

rockwotj Mar 14, 2024

dotnwat Mar 14, 2024

rockwotj Mar 14, 2024

dotnwat Mar 14, 2024

rockwotj Mar 14, 2024

rockwotj commented Mar 14, 2024

rockwotj commented Mar 14, 2024

rockwotj commented Mar 14, 2024

oleiman left a comment

		When writing C++ in WebAssembly, note that certainly language features are not supported.
		Noteably C++ exceptions, `setjmp`/`longjmp`, threads and networking is not supported.

transform-sdk: experimental C++ support #16229

transform-sdk: experimental C++ support #16229

Conversation

rockwotj commented Jan 22, 2024 • edited

Backports Required

Release Notes

vbotbuildovich commented Jan 22, 2024 • edited

andrewhsu left a comment

Choose a reason for hiding this comment

rockwotj commented Jan 28, 2024

oleiman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rockwotj commented Mar 14, 2024

rockwotj commented Mar 14, 2024

rockwotj commented Mar 14, 2024

oleiman left a comment

Choose a reason for hiding this comment

rockwotj commented Jan 22, 2024 •

edited

vbotbuildovich commented Jan 22, 2024 •

edited