[Sprint: 38] XD-2339a KafkaConsumer configs as module options #1288
Conversation
ilayaperumalg
commented
Nov 15, 2014
ilayaperumalg
commented
- Remove external config properties for kafka consumer
- Add module options for all the possible kafka consumer configurations
ilayaperumalg
changed the title
- Remove external config properties for kafka consumer - Add module options for all the possible kafka consumer configurations
ilayaperumalg
force-pushed
the
XD-2339a
branch
from
1efdf9c
to
01ceb87
Compare
ilayaperumalg
changed the title
ilayaperumalg
changed the title
|
|
||
| private int refreshLeaderBackOff = 200; | ||
|
|
||
| private String autoOffsetReset = "largest"; |
There was a problem hiding this comment.
Is the switch here from "smallest" to "largest" intentional, in order to align the defaults within Kafka?
I think that such an alignment is generally desirable, but in this case, it is up to us to decide how we want new sources to behave (i.e. start at the end of the log or at the beginning of the log), and "smallest" for new sources would be a better strategy, IMO.
There was a problem hiding this comment.
yeah, it is align with the default values that Kafka suggests. I can change it to "smallest". Could you elaborate why "smallest" value is desirable in this case?
There was a problem hiding this comment.
I think that in this case we should align with https://github.com/spring-projects/spring-integration-kafka/blob/master/src/main/java/org/springframework/integration/kafka/core/KafkaConsumerDefaults.java
In particular, I find that "smallest" is better from a business standpoint because it causes new consumer groups to start at the beginning of the log, which is rather deterministic, rather than at the end, which isn't. This goes better with the semantic that a new source with a new groupId is a different view on the same set of data. Of course, these are just defaults, so users can override these values as they feel necessary.
There was a problem hiding this comment.
Also, the value used to be "smallest" prior to 0.8, and it's been changed, in order to accomodate migration from 0.7 to 0.8 - I don't think that motivation applies to us.
There was a problem hiding this comment.
The default value for consumer timeout in Kafka SI is set to use 5000 ms and the actual default is "-1" which makes the consumer wait indefinitely. Since we have module lifecycle, I believe setting it to "-1" makes sense. What do you think?
There was a problem hiding this comment.
As per our discussion, we should avoid thread blocking.
|
Merged 1d8a1fa |
