-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Issue 9480][pulsar-io] add option for auto.offset.reset to kafka source #9482
Conversation
@BewareMyPower Could you have a look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you add the sample config to kafkaSourceConfig.yaml
and verify it in loadFromYamlFileTest
?
Sample config added to kafkaSourceConfig.yaml and verified in loadFromYamlFileTest |
Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/pulsarbot rerun-failure-checks |
/pulsarbot rerun-failure-checks |
/pulsarbot rerun-failure-checks |
1 similar comment
/pulsarbot rerun-failure-checks |
…rce (#9482) Fixes #9480 ### Motivation The kafka source sets auto.offset.reset to "earliest". This means all old messages from kafka are produced to pulsar. Often is it desirable to start form the present location "latest". The option is set after the user config has been loaded so it cannot be changed: [source code link](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L87) ### Modifications Added an autoOffsetReset option to KafkaSourceConfig (cherry picked from commit 9c2b081)
Fixes #9480
Motivation
The kafka source sets auto.offset.reset to "earliest". This means all old messages from kafka are produced to pulsar. Often is it desirable to start form the present location "latest".
The option is set after the user config has been loaded so it cannot be changed:
source code link
Modifications
Added an autoOffsetReset option to KafkaSourceConfig
Verifying this change
(Please pick either of the following options)
This change added tests and can be verified as follows:
Added to unit test and verified behaviour.
Does this pull request potentially affect one of the following parts:
If
yes
was chosen, please highlight the changesDocumentation