-
Notifications
You must be signed in to change notification settings - Fork 492
Fix Kinesis source panic on resharding #5912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
cc @guilload |
cf51110 to
2b8e002
Compare
5f60916 to
4774727
Compare
|
I fixed the lint failure. |
guilload
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good. The source could use a logging statement when we spawn a new shard consumer with stream name, shard_id, start position.
4774727 to
3690a7d
Compare
|
@guilload Thanks for your review. I've added the logging statement you mentioned. |
|
Awesome. I will merge. Thanks, @earlbread! |
|
@guilload Thank you! |
Description
Currently, the Kinesis source would panic when encountering resharding scenarios (shard splits/merges) because it attempted to create duplicate consumers for the same shard.
This occurred when:
This replaces the assertion with a check that safely skips duplicate consumer creation.
Fixes #5896
How was this PR tested?
test_kinesis_source_handles_resharding_with_splittest