Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dekaf: introduce new materialization endpoint type #1622

Closed
jshearer opened this issue Sep 10, 2024 · 0 comments
Closed

dekaf: introduce new materialization endpoint type #1622

jshearer opened this issue Sep 10, 2024 · 0 comments
Assignees

Comments

@jshearer
Copy link
Contributor

jshearer commented Sep 10, 2024

The intent of this task is to flesh out the framing for how Dekaf is used. Currently, it works by exposing all of the collections you have read access to as Kafka topics. This isn't ideal for a few reasons:

  • Stats/logs/billing
  • Bindings/Projections
  • Config
  • UX

We already have a bunch of design and infrastructure built to support creating and configuring tasks that pull data out of Flow collections: materializations. So, the idea is to frame Dekaf usage as another kind of materialization. Currently, we support two types of materializations: Local and Connector. The idea is to add a third type Dekaf.

This stretches the abstraction a bit, as at present the intent of a materialization is to (describe how to) actively run a task that performs the work of materialization, whereas a Dekaf-type materialization won't directly start any work and instead just act as a vehicle for configuring actual Dekaf, but AFAICT it doesn't introduce any direct contradictions and does feel like it elegantly gets us all of the presently missing features we want.

In practice, this will look something like this:

  • Add a new MaterializationEndpoint::Dekaf variant. This will contain whatever configuration is specific to this dekaf task such as the strict_topic_names flag.
  • When a materialization using this endpoint type gets activated, it will still create ops collections for logs and stats, but it won't start up any containers.
  • Dekaf will be updated to take the name of a materialization as part of its config input provided in the username field. Then, instead of advertising all available collections, it will advertise all of the enabled bindings for that materialization instead.
  • Dekaf will directly write logs and stats to the ops collections of its configured materialization task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants