Allow further configuration of topics

[jhill072]

Problem
I would like to control more topic configurations than the currently exposed partitions and replication factor, such as retention period, cleanup type, and potentially 3rd-party configurations like confluent.*

Persona:
Which persona is this feature for?

• System Operator

Exit Criteria
I can set properties such as "cleanup.policy", "retention.ms", and "confluent.tier.enable" for the topic backed by the Broker.

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

In my mind, this seems really close to the existing config map for a topic, but with more configurations, so I would picture at most a week?

Additional context (optional)
Add any other context about the feature request here.

Originally requested here, but deserves its own issue.

[pierDipi]

/triage accepted
/area broker
/area control-plane
/kind api-change

[pierDipi]

As always, contributions are welcome and I'm happy to provide guidance for the implementation!

/good-first-issue

[knative-prow-robot]

@pierDipi:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

As always, contributions are welcome and I'm happy to provide guidance for the implementation!

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sub-zero7]

Hello, what is a default retention period for managed and unmanaged topics (when kafka.eventing.knative.dev/external.topic is set)? Are default.topic.partitions and default.topic.replication.factor ignored in unmanaged topic scenario? Thanks

[pierDipi]

Hello, what is a default retention period for managed and unmanaged topics (when kafka.eventing.knative.dev/external.topic is set)? Are default.topic.partitions and default.topic.replication.factor ignored in unmanaged topic scenario? Thanks

Correct, in the unmanaged case, we ignore replication factor and num partitions configurations, in the managed case, you need to specify them in a ConfigMap, we ship a default ConfigMap in the knative-eventing namespace with these values:
https://github.com/knative-sandbox/eventing-kafka-broker/blob/7678c99b6f802f16367a7b3fdcff2f06c5d485e4/control-plane/config/eventing-kafka-broker/100-broker/100-kafka-broker-configmap.yaml#L25-L27

[sub-zero7]

Thanks for the clarification. We are also currently thinking about topic retention period. For example in alternative option https://github.com/knative-sandbox/eventing-kafka/blob/release-1.5/pkg/channel/distributed/README.md it's configurable and has a default value of retentionDuration: PT168H. As this issue mentions eventing-kafka-broker currently does not allow to configure that, so I would like to understand for how long time events will be persisted in a broker in managed and unmanaged topic cases (for example, to plan event replay scenarios).

[pierDipi]

in the managed case it's using whatever is the Kafka cluster default, in the unmanaged case, you can specify it when you're creating the topic or if unspecified it will use the Kafka cluster default.

[vishal-chdhry]

/assign

[vishal-chdhry]

@debasishbsws go ahead!

[debasishbsws]

/assign

[Leo6Leo]

@debasishbsws hey debasish are u still working on this?

[debasishbsws]

@Leo6Leo No, I am not any more, you can take it over.

[Cali0707]

/assign