Self-Balancing Knative Kafka Broker partitions

[pierDipi]

Problem

Creating a Knative Kafka Broker requires developers to specify the number of partitions the backing Kafka topic should have, however, this is not an easy decision to make and it requires planning based on the expected load the Knative Broker could receive.
This project aims to remove this configuration setting by having an autoscaler that is responsible to add or remove partitions (within bounds) based on the effective load the Knative Kafka Broker receives while preserving ordered and unordered delivery features for Triggers.

Persona:
Which persona is this feature for?

• Event producer

Exit Criteria

A Knative Kafka Broker can be created without setting the number of partitions and the number of partitions for the topic backing the Knative Kafka Broker increases or decreases based on the observed load received.

Additional context (optional)

• https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/
• https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/#configure-a-kafka-broker
• ordered and unordered delivery features for Triggers

[dharmicksai]

Hello 👋🏽 ,
I am Dharmick, a final-year UG student from India.
This problem Statement looks interesting !!!

I have previously worked with Kafka, Kubernetes, and Scala.
The number of partitions in a topic limits the number of consumers from a consumer group,
So scaling the number of partitions based on load is beneficial.

I found one method for modifying the number of partitions in kafka docs, but this does not preserve the ordering
Can I get guidance on how the number of partitions can be modified by preserving the order of events?

[pierDipi]

Hi @dharmicksai, the ordering is a real challenge, however, the Knative Kafka Broker is not always bound to a specific topic which means we can "migrate" producers and consumers to a (different) topic while also keeping ordering guarateed.

[Abhishek357]

Hi @dharmicksai, the ordering is a real challenge, however, the Knative Kafka Broker is not always bound to a specific topic which means we can "migrate" producers and consumers to a (different) topic while also keeping ordering guarateed.

hey, @pierDipi in the above approach why are you suggesting to migrate producer and consumer to different topic? idealy to solve this issue we should implement some autoscaler approach to increase/decrease the number of partition for a same topic?

[anurag-sachan]

Hey @pierDipi,
I am new to Open Source, I have looked into the issue, I understood about the configuration & ordering stated in kafka docs.
I wanted to know, Is number of partitions for the topic is same as = concurrency for auto-scaling ?

[wasup-yash]

Hey, it will be better to add [LFX-Mentorship] as a label as it will be part of the LFX-mentorship program and this is an upstream issue in the project defined as Self-Balancing Knative Kafka Broker partitions , and I would also like to work on this project as I have worked with Kubernetes and GOlang and willing to learn the Kafka on the way with the project ... should I look at some issues before jumping into this? so please recommend it.

[rukundob451]

Hello @pierDipi,
Thank you for bringing up the problem related to creating a Knative Kafka Broker. The current requirement of specifying the number of partitions for the backing Kafka topic poses challenges as it requires careful planning based on the expected load that the Knative Broker might receive.

To address this issue, the proposed solution aims to eliminate the need for manually configuring the number of partitions. Instead, an autoscaler will be implemented to dynamically manage the partition count within predefined bounds. The autoscaler will monitor the effective load received by the Knative Kafka Broker and adjust the number of partitions accordingly, ensuring efficient scalability.

Regarding the target persona for this feature, it primarily benefits the "Event producer" persona. As an event producer, they can utilize the Knative Kafka Broker without the burden of explicitly specifying partition numbers. The autoscaler will handle the partition management, allowing the event producer to focus on their core tasks.

To meet the exit criteria for this feature, I believe the following conditions should be fulfilled:

It should be possible to create a Knative Kafka Broker without the need to set the number of partitions explicitly.
The number of partitions for the Kafka topic backing the Knative Kafka Broker should dynamically increase or decrease based on the observed load received by the broker.
For more detailed information and guidance, I recommend referring to the provided resources:

Knative documentation on Kafka Broker: Link 1
Specific configuration details for the Kafka Broker: Link 2

Kindly wanted to know what you think about this?

[smilingprogrammer]

Thanks @rukundob451

[AllMight2099]

Hi @pierDipi
Is this issue still open? I'd like to try my hand at it.