[pulsar-java-client] Decouple partition switch frequency with maxPublishDelay in producer configuration #5788
Conversation
| * @see #messageRoutingMode(MessageRoutingMode) | ||
| * @see #batchingMaxPublishDelay(long, TimeUnit) | ||
| */ | ||
| ProducerBuilder<T> batchingPartitionSwitchFrequency(int frequency); |
There was a problem hiding this comment.
It is better to add both time interval and time unit in the method, by following the convention of other time related settings.
There was a problem hiding this comment.
The current frequency is used to compute switch delay in frequency * maxBatchDelayMicros. Do you mean I should make it a standalone variable? Then if users didn't set this variable, or the user is setting a variable less than maxBatchDelay, I need to use maxBatchDelay in round-robin partitioner?
There was a problem hiding this comment.
Oh I misread the code. I see it now. Then that's fine to me.
| private MessageRouter customMessageRouter = null; | ||
|
|
||
| private long batchingMaxPublishDelayMicros = TimeUnit.MILLISECONDS.toMicros(1); | ||
| private int batchingPartitionSwitchFrequencyByPublishDelay = 10; |
There was a problem hiding this comment.
Add "Micros" to the name so people know the time unit directly by reading the name.
codelipenghui
left a comment
codelipenghui
left a comment
There was a problem hiding this comment.
The change looks good me, is it possible implement in RoundRobinPartitionMessageRouterImpl? Looks the batching partition switch frequency only works in RoundRobin mode. Please considering whether this configuration should in ProducerBuilder.
I think we are exposing the capability of changing partition switch frequency to users? If so, the user conf in |
|
@sijie I have discussed with yijie, i'm not sure use a custom router is reasonable here. we can support setting frequency in RoundRobinPartitionMessageRouterImpl and user use it like: Which can reduces |
|
@codelipenghui that works for me. |
|
run java8 tests |
|
run java8 tests. |
3 participants
Motivation
Currently, while using round-robin message router, messages would be route to different partitions at the
maxPublishDelayboundaries. However, when GC occurs, the output partition would change frequently and results in small batches.This PR introduces a conf field setting the round-robin partition switch frequency:
batchingPartitionSwitchFrequencyByPublishDelay. With this PR, the partition switch interval could be set asmaxPublishDelay*frequency.Modifications
batchingPartitionSwitchFrequencyByPublishDelayinProducerConfiguratiionData.javaVerifying this change
This change is already covered by existing tests.
Does this pull request potentially affect one of the following parts:
Documentation