Kafka scaler: limit number of replicas to the number of partitions with lag.

[alexakra]

Proposal

I would like to have an option to limit number of replicas to the number of partitions with lag > 0 only.

Use-Case

When using Kafka key, the amount of records per key (partition) not always could be distributed evenly, and in some cases only some partitions have a lag while most of the partitions have no lag.
A special case is when only one partition has a lag which could easily scale to the max allowed number of replicas while all the consumers will be idle except one. In such case there is no need to scale over one replica.
In a general case, I would like to limit scaling to min(max allowed number of replicas as of today, number of partitions with lag > 0). Even though, the partitions could be distributed unevenly between consumers, sometimes it is more efficient way to allocate resources rather than scaling idle consumers.

Anything else?

No response

[zroubalik]

Sounds reasonable. My Kafka knowledge is limited, but could you consider following example:

partitionA - lag = 0
partitionB - lag = 0
partitionC - lag = 100

and we have one active consumer subscribed to partitionA. What would happen if your proposed change is enabled? Is the consumer smart enough the re-subscribe to partitionC in this case?

[alexakra]

Usually a consumer subscribes to topic (not to partition), i.e. one active consumer will be subscribed to all partitions.
When number of consumer (the same consumer group) replicas changes, then Kafka broker will split/join the partitions between all the active consumers and rebalancing process will be triggered.

[zroubalik]

ack, thanks for the clarification.

Are you willing to contribute this feature?

[alexakra]

Unfortunately, I am not golang developer.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[DmitrySenin]

So was it implemented?

[zroubalik]

I was not. It was autoclosed, are you interested in contributing this @DmitrySenin ?

[DmitrySenin]

@zroubalik , I would love to try

[stale]

This issue has been automatically closed due to inactivity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[kamilglod]

I'm interested in this feature as well - it could be useful for use case where you're manually assigning partition by producer - to e.g. not block all partitions with huge events publication. In that case you end up with only some partitions with lag.