Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch Listener - Support Zombie Fencing #1278

Closed
garyrussell opened this issue Oct 18, 2019 · 0 comments · Fixed by #1280
Closed

Batch Listener - Support Zombie Fencing #1278

garyrussell opened this issue Oct 18, 2019 · 0 comments · Fixed by #1280
Assignees
@garyrussell
Labels
type: enhancement
Milestone
2.3.2

Comments

@garyrussell
Copy link
Contributor

garyrussell commented Oct 18, 2019 •
edited

Currently, zombie fencing is not supported for a batch listener because a batch can contain records from multiple partitions and zombie fencing requires a transactional.id based on the group/topic/partition.

Add an option to support zombie fencing by processing ConsumerRecords one partition at a time - i.e. pass a list of ConsumerRecord to the listener for each partition received, instead of a single list containing all records received.

A transaction will then span each partition and the proper transactional.id can be used for each sub batch.

FilteringBatchMessageListenerAdapter will need to create a new list if any records are filtered because we get an unmodifiable list.
@garyrussell garyrussell added this to the 2.4.M1 milestone Oct 18, 2019
@garyrussell garyrussell self-assigned this Oct 18, 2019
@garyrussell garyrussell modified the milestones: 2.4.M1, 2.3.2 Oct 21, 2019
garyrussell added a commit to garyrussell/spring-kafka that referenced this issue Oct 21, 2019
@garyrussell
spring-projectsGH-1278: Zombie Fencing with Batch Listener
d8b940e 
Resolves spring-projects#1278

Support transaction per partition with Batch Listeners so the `transactional.id`
is tied to the group/topic/partition.
garyrussell added a commit to garyrussell/spring-kafka that referenced this issue Oct 21, 2019
@garyrussell
spring-projectsGH-1278: Zombie Fencing with Batch Listener
4634b28 
Resolves spring-projects#1278

Support transaction per partition with Batch Listeners so the `transactional.id`
is tied to the group/topic/partition.
@garyrussell garyrussell mentioned this issue Oct 21, 2019
Merged
artembilan pushed a commit that referenced this issue Oct 21, 2019
@garyrussell @artembilan
GH-1278: Zombie Fencing with Batch Listener
4855751 
Resolves #1278

Support transaction per partition with Batch Listeners so the `transactional.id`
is tied to the group/topic/partition.

* Fix javadoc
garyrussell added a commit that referenced this issue Oct 28, 2019
@garyrussell
GH-1278: Add filtered sub batch test
2f734a7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@garyrussell garyrussell

Labels
type: enhancement
Projects
None yet
Milestone
2.3.2
Development

Successfully merging a pull request may close this issue.

GH-1278: Zombie Fencing with Batch Listener garyrussell/spring-kafka
1 participant
@garyrussell