-
Notifications
You must be signed in to change notification settings - Fork 181
KafkaMessageSource and concurrency #211
Comments
putting this:
inside the synchronized block should help with the ConcurrentModificationException |
@bruto0 , It's very likely you are fully on board with what's going on. Thanks |
Moving a curly brace a few lines down isn't much of a PR :) |
While thread-safety should be fixed; acking like that probably won't do what you need. If you hand the message off to another thread, the acks might be processed out of order, which is probably not what you want. With Kafka, only a log offset is committed, each message is not "ack'd" per se. So if the record at offset 2 is ack'd first, the record at offset 1 is also "ack'd". Worse, when offset 1 is later committed (and let's say the app is restarted), you will get the record at offset 2 again. |
It does what I need :) I do know that Kafka is not a MQ and seeking backwards means replayed messages - it's ok in my case. My app is a hack job, but we do need the rewind (seek) option and record processing outside of consumer thread so I made KMS work for me rather than write something similar on top of spring-kafka myself :) |
Actually, I forgot; we added code to defer "future" acks and only commit them when the "gaps" are filled in. I am fixing the concurrency now... |
Fixes spring-attic#211 Use `ConcurrentHashMap` with synchronized set values to avoid concurrency issues when acks are processed on multiple threads.
Fixes #211 Use `ConcurrentHashMap` with synchronized set values to avoid concurrency issues when acks are processed on multiple threads.
Fixes #211 Use `ConcurrentHashMap` with synchronized set values to avoid concurrency issues when acks are processed on multiple threads. # Conflicts: # src/main/java/org/springframework/integration/kafka/inbound/KafkaMessageSource.java
Fixes spring-attic/spring-integration-kafka#211 Use `ConcurrentHashMap` with synchronized set values to avoid concurrency issues when acks are processed on multiple threads.
Fixes spring-attic/spring-integration-kafka#211 Use `ConcurrentHashMap` with synchronized set values to avoid concurrency issues when acks are processed on multiple threads.
If you disable autoack in AcknowledgmentCallbackFactory and ack/requeue offsets in a different thread from the one poller is operating on, all sorts of things become possible here under sufficient load:
ConcurrentModificationException, NoSuchElementException
Also, acknowledged flag is set in the finally block of KafkaAckCallback.acknowledge unconditionally, although in case of ConcurrentModificationException you likely want to try ack'ing the offset again
And while we're at it, it'd be nice to have an ability to ack a bunch of offsets belonging to the same partition without sync'ing on the consumerMonitor for each one, but you'd have to expose KafkaMessageSource from the inbound channel adapter for that to work
The text was updated successfully, but these errors were encountered: