KAFKA-20300: Add broker configs for assignment batching and offload#21730
KAFKA-20300: Add broker configs for assignment batching and offload#21730dajac merged 2 commits intoapache:trunkfrom
Conversation
…load
Add group.{consumer,share,streams}.assignment.interval.ms config options
to control the delay between assignment calculation. These config
options are dynamic at the broker level.
Add group.{consumer,share,streams}.{min,max}.assignment.interval.ms
config options to limit the
group.{consumer,share,streams}.assignment.interval.ms config values.
Add group.{consumer,share,streams}.assignor.offload.enable config
options to control whether assignment calculation is offloaded to a
group coordinator background thread. These config options are dynamic at
the broker level.
github-actions
bot
added
triage
| newProps ++= staticBrokerConfigs | ||
| overrideProps(newProps, dynamicDefaultConfigs) | ||
| overrideProps(newProps, dynamicBrokerConfigs) | ||
| GroupCoordinatorConfig.clampDynamicConfigs(newProps.asJava) |
There was a problem hiding this comment.
It's analogous to the group config clamping in KIP-1240. Without it, all broker-level dynamic configs will not be applied when a broker is starting and the dynamic assignment.interval.ms is now outside of the allowed range.
There was a problem hiding this comment.
I see... Thanks for clarifying. I don't really like having this here though. Have you considered alternative approaches?
There was a problem hiding this comment.
At minimum, I think that we should have a clampDynamicConfigs methods in KafkaConfig so we avoid the coupling with GroupCoordinatorConfig here. It would also give a hit to folks adding configs to KafkaConfig.
I was also considering whether we should just introduce a new argument to the KafkaConfig, say clampDynamicConfigs. I am not sure whether it is better.
There was a problem hiding this comment.
Defining KafkaConfig.clampDynamicConfigs makes sense.
And then we can either call it here or in the KafkaConfig constructor. If it's the latter we have to thread through an extra bool through processReconfiguration. Both threading the bool vs not threading are about the same for me.
|
@squah-confluent Could you please create a dedicated jira for this one? |
squah-confluent
changed the title
|
@dajac I filed KAFKA-20300 for this. |
2 participants
Add group.{consumer,share,streams}.assignment.interval.ms config options
to control the delay between assignment calculation. These config
options are dynamic at the broker level.
Add group.{consumer,share,streams}.{min,max}.assignment.interval.ms
config options to limit the
group.{consumer,share,streams}.assignment.interval.ms config values.
Add group.{consumer,share,streams}.assignor.offload.enable config
options to control whether assignment calculation is offloaded to a
group coordinator background thread. These config options are dynamic at
the broker level.
Reviewers: David Jacot djacot@confluent.io