Support incremental cooperative rebalancing

[chiawendt]

Problem: Stop-the-world rebalances

When scaling consumers up or down, a rebalance is triggered. Depending on the partition assignment strategy, this can pause all consumers for about 30 seconds, leading to temporary downtime. This “stop-the-world” effect makes scaling disruptive in production environments.

Proposed solutions

1. KIP-429 CooperativeStickyAssignor: Introduces incremental cooperative rebalancing by reassigning partitions gradually, so not all consumers are paused at once.
2. KIP-848 New consumer rebalancing protocol: Moves partition assignment decisions to the broker and provides incremental rebalancing by default. Shipped with Kafka 4.0, it effectively supersedes KIP-429.

[mcollina]

Would you like to implement KIP-848 or both?

[chiawendt]

I am interested in implementing it, but since this is a big task I might not be able to complete it in my free time.
Yes I am working on it.