SAMZA-2073: Do not commit the task offsets when shutting down the SamzaContainer. #884
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@prateekm Can you take a look since you have more context. |
shanthoosh
force-pushed
the
remove_task_commit_during_shutdown
branch
from
5cb3faf
to
3f37ecd
Compare
prateekm
reviewed
Jan 16, 2019
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
Show resolved
Hide resolved
prateekm
approved these changes
Jan 16, 2019
|
@prateekm Thanks for the quick review. |
Zhangyx39
pushed a commit
to Zhangyx39/samza
that referenced
this pull request
Apr 3, 2019
…zaContainer. SAMZA-1489 added support for committing the offsets of all the tasks when shutting down a SamzaContainer. Other components in samza such as a CheckpointListener's aren't developed to account for possibility of commit after the consumers are stopped. This unnecessarily results in a unclean shutdown of a samza standalone processor during the rebalancing phase. Here're the sample logs: ``` apache.samza.container.SamzaContainer129d533c to shutdown. 2019/01/15 02:38:09.738 INFO [SamzaContainer] [Samza StreamProcessor Container Thread-0] [hello-brooklin-task] [] Shutting down SamzaContainer. 2019/01/15 02:38:09.738 INFO [SamzaContainer] [Samza StreamProcessor Container Thread-0] [hello-brooklin-task] [] Shutting down consumer multiplexer. 2019/01/15 02:38:09.741 INFO [KafkaProducer] [hello-brooklin-task-i001-auditor] [hello-brooklin-task] [] [Producer clientId=hello-brooklin-task-i001-auditor, transactionalId=nullClosing the Kafka producer with timeoutMillis = 9223370489334886066 ms. 2019/01/15 02:38:09.748 INFO [SamzaContainer] [Samza StreamProcessor Container Thread-0] [hello-brooklin-task] [] Shutting down task instance stream tasks. 2019/01/15 02:38:09.748 INFO [SamzaContainer] [Samza StreamProcessor Container Thread-0] [hello-brooklin-task] [] Shutting down timer executor 2019/01/15 02:38:09.749 INFO [SamzaContainer] [Samza StreamProcessor Container Thread-0] [hello-brooklin-task] [] Committing offsets for all task instances 2019/01/15 02:38:09.753 ERROR [SamzaContainer] [Samza StreamProcessor Container Thread-0] [hello-brooklin-task] [] Caught exception/error while shutting down container. java.lang.IllegalStateException: This consumer has already been closed. at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1735) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1214) at com.linkedin.kafka.liclients.consumer.LiKafkaConsumerImpl.commitOffsets(LiKafkaConsumerImpl.java:311) at com.linkedin.kafka.liclients.consumer.LiKafkaConsumerImpl.commitSync(LiKafkaConsumerImpl.java:282) at com.linkedin.brooklin.client.BaseConsumerImpl.commit(BaseConsumerImpl.java:136) ``` Since the final commit is not critical, it will be better to not do it as a part of the SamzaContainer shutdown sequence. Author: Shanthoosh Venkataraman <spvenkat@usc.edu> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#884 from shanthoosh/remove_task_commit_during_shutdown
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SAMZA-1489 added support for committing the offsets of all the tasks when shutting down a SamzaContainer. Other components in samza such as a CheckpointListener's aren't developed to account for possibility of commit after the consumers are stopped.
This unnecessarily results in a unclean shutdown of a samza standalone processor during the rebalancing phase. Here're the sample logs:
Since the final commit is not critical, it will be better to not do it as a part of the SamzaContainer shutdown sequence.