Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Wasm Golang Transform SDK #12322

Merged
merged 4 commits into from
Aug 10, 2023

Commits on Jul 19, 2023

  1. Initial transform SDK module boilerplate

    Add the Apache 2.0 License as developers will be linking this into their
    custom Wasm transforms.
    
    Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
    rockwotj committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    bb07d58 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. Add internal rwbuf package

    This package is a resizable buffer that tracks a reader and writer index
    and is generally easier to use than bytes.Buffer for what we're trying
    to do when reading/writing across the Wasm FFI boundary.
    
    Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
    rockwotj committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    981e751 View commit details
    Browse the repository at this point in the history
  2. Initial Wasm transform SDK

    This is the initial SDK for Wasm transforms.
    
    We have an ABI that we expect from the broker, and the broker expects
    from this SDK. We mostly pass buffers back and forth across the ABI
    boundary, and in these buffers are Kafka wire format serialized records.
    
    We operate on single records at a time so that we use as little memory as
    possible within the Wasm runtime (there is very little overhead to the
    Wasm function calls).
    
    Lastly, the ABI is stubbed out on platforms other than Wasm so you can
    test with the SDK without running it in Wasm.
    
    Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
    rockwotj committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    4328702 View commit details
    Browse the repository at this point in the history
  3. Add documentation and examples

    We add some simple documentation, and three examples.
    
    Mirror: the simplest transform possible, just copy data to another
    topic.
    
    Regexp Filter: Shows off main() setup and environment variable
    configuration
    
    Transcoding: Hopefully something a little more real world without
    pulling in Avro or some third party library.
    
    Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
    rockwotj committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    8527899 View commit details
    Browse the repository at this point in the history