-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EventHubs stops processing outgoing messages #17568
Comments
@conniey could you please follow up? |
Hi, Exceptions which we see most often:
Can't tell which exactly is the real issue. Sometimes it works for days without any problem, sometimes it fails after 1 hour and never recovers. Issuing client.stop followed by client.start doesn't help. Neither creating new EventProcessorClient does. Only full application restart is proven to "fix" the problem. We use Azure Kubernetes to run the application. OS is Ubuntu-18, Java version 8. BR |
We are seeing similar issues too. When we scale down the number of PODS for Consumers, the remaining Consumers stop processing Events and when they scale it back up it starts processing. Also all events between Scale Down and Scale up are lost. Restarting the PODS post scale down helps to restart the processing. I checked on com.microsoft.azure:spring-cloud-azure-eventhubs-stream-binder version and its - 1.2.8 We also have seen below intermittent errors in Consumers. Any pointers on how to resolve this will help. |
Closed as a duplicate of #18070 |
Describe the bug
We have project on spring boot and
using
com.azure
azure-messaging-eventhubs
5.3.1
and EventProcessorClient api to listen messages from topic using eventProcessor.
We are seeing that time to time listener stops listening from the topic
even though our application was up.
Blue line in graph showing msgs incoming
Orange line showing outgoing msgs
We can see there are time windows mentioned below where we have only incoming msgs
but no outgoing msgs.
1:34 to 12:15 incoming msgs but no outgoing msgs
12:33 to 1:22 incoming msgs but no outgoing msgs
similarly other time windows.
In one scenario we saw the listener stops listening msgs from topic but continously updating ownership last modified time in storagev2 checkpoint blob storage.
Also at the same time few partition were listening msgs while some stopped listening.
Also when we restarted the app listener started listening messagess
Exception or Stack Trace
We have different exception in our logs occurring very frequently dont know if these are actually the rootcause.
Exception occurred when handling event in super.
Error occurred in XXXXX partition processor for partition 8, com.azure.core.amqp.exception.AmqpException: The connection was inactive for more than the allowed 240000 milliseconds and is closed by container '4ddce78e588743768330a20fc913f4f5_G14'., errorContext[NAMESPACE: ehns-control-tower-prod-scus-1.servicebus.windows.net, PATH: YYYYY/ConsumerGroups/XXXX/Partitions/8, REFERENCE_ID: 8_79eeb9_1605218761061, LINK_CREDIT: 473]
Error occurred in partition processor for partition 12, java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 60000ms in 'takeUntil' (and no fallback has been configured)
Error occurred in partition processor for partition 17, com.azure.core.amqp.exception.AmqpException: New receiver 'nil' with higher epoch of '0' is created hence current receiver 'nil' with epoch '0' is getting disconnected. If you are recreating the receiver, make sure a higher epoch is used. TrackingId:9424c7270001399b000071e85fadc19f_G27_B27, SystemTracker:ehns-control-tower-prod-scus-1:eventhub:XXXXX~18431|YYYY, Timestamp:2020-11-12T23:14:59, errorContext[NAMESPACE: ehns-control-tower-prod-scus-1.servicebus.windows.net, PATH: XXXXX/ConsumerGroups/YYYYYY/Partitions/17, REFERENCE_ID: 17_0989fc_1605218311041, LINK_CREDIT: 447]
The text was updated successfully, but these errors were encountered: