You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The intent of this task is to flesh out the framing for how Dekaf is used. Currently, it works by exposing all of the collections you have read access to as Kafka topics. This isn't ideal for a few reasons:
Stats/logs/billing
Bindings/Projections
Config
UX
We already have a bunch of design and infrastructure built to support creating and configuring tasks that pull data out of Flow collections: materializations. So, the idea is to frame Dekaf usage as another kind of materialization. Currently, we support two types of materializations: Local and Connector. The idea is to add a third type Dekaf.
This stretches the abstraction a bit, as at present the intent of a materialization is to (describe how to) actively run a task that performs the work of materialization, whereas a Dekaf-type materialization won't directly start any work and instead just act as a vehicle for configuring actual Dekaf, but AFAICT it doesn't introduce any direct contradictions and does feel like it elegantly gets us all of the presently missing features we want.
In practice, this will look something like this:
Add a new MaterializationEndpoint::Dekaf variant. This will contain whatever configuration is specific to this dekaf task such as the strict_topic_names flag.
When a materialization using this endpoint type gets activated, it will still create ops collections for logs and stats, but it won't start up any containers.
Dekaf will be updated to take the name of a materialization as part of its config input provided in the username field. Then, instead of advertising all available collections, it will advertise all of the enabled bindings for that materialization instead.
Dekaf will directly write logs and stats to the ops collections of its configured materialization task.
The text was updated successfully, but these errors were encountered:
The intent of this task is to flesh out the framing for how Dekaf is used. Currently, it works by exposing all of the collections you have read access to as Kafka topics. This isn't ideal for a few reasons:
We already have a bunch of design and infrastructure built to support creating and configuring tasks that pull data out of Flow collections: materializations. So, the idea is to frame Dekaf usage as another kind of materialization. Currently, we support two types of materializations:
Local
andConnector
. The idea is to add a third typeDekaf
.This stretches the abstraction a bit, as at present the intent of a materialization is to (describe how to) actively run a task that performs the work of materialization, whereas a
Dekaf
-type materialization won't directly start any work and instead just act as a vehicle for configuring actual Dekaf, but AFAICT it doesn't introduce any direct contradictions and does feel like it elegantly gets us all of the presently missing features we want.In practice, this will look something like this:
MaterializationEndpoint::Dekaf
variant. This will contain whatever configuration is specific to this dekaf task such as thestrict_topic_names
flag.The text was updated successfully, but these errors were encountered: