Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Host suddenly stops receiving new messages without recovering #377

Closed
royyeah opened this issue Sep 3, 2018 · 9 comments
Closed

Host suddenly stops receiving new messages without recovering #377

royyeah opened this issue Sep 3, 2018 · 9 comments
Assignees
@SreeramGarlapati

Comments

@royyeah
Copy link

royyeah commented Sep 3, 2018 •
edited
Loading

Actual Behavior

I am using an EventProcessorHost with basic setup as explained here. Recently we ran into problems where the host stops receiving new messages because it did not recover correctly. When we restarted the application it continued processing again.

Expected Behavior

I think the warnings below are the result of event hub partitions being moved across nodes. I would expect the event processor host to automatically recover from this.

Is this fixed in azure-eventhubs 1.1.0? I see this contains bugfixes #372 and #373 which sound like potential fixes.

Versions

  • OS platform and version: Debian 8
  • Maven package version or commit ID:
    • azure-eventhubs: 1.0.2
    • azure-eventhubs-eph: 2.0.1

More details

Log excerpt:

2018-09-02 08:32:45,500 WARN  [pool-2-thread-9] c.m.a.e.i.ReceivePump - Receive pump for partition (8) exiting after receive exception java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:cd407ddfef2847d0a54fafa43b6c42bc_B13, SystemTracker:SharedConnectionListener, Timestamp:9/2/2018 8:32:45 AM TrackingId:2a1dd72a1af240b295ecaa259ca8928a_G1, SystemTracker:gateway5, Timestamp:9/2/2018 8:32:45 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/8, REFERENCE_ID: cf942a_928a_G1_1535877165274, PREFETCH_COUNT: 300, LINK_CREDIT: 0, PREFETCH_Q_LEN: 0]
2018-09-02 08:32:45,500 WARN  [pool-2-thread-9] c.m.a.e.PartitionPump - host salvador-to-cassandra-758ed9cf-722d-4674-b58d-a68803f15738: 8: EventHub client error: java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:cd407ddfef2847d0a54fafa43b6c42bc_B13, SystemTracker:SharedConnectionListener, Timestamp:9/2/2018 8:32:45 AM TrackingId:2a1dd72a1af240b295ecaa259ca8928a_G1, SystemTracker:gateway5, Timestamp:9/2/2018 8:32:45 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/8, REFERENCE_ID: cf942a_928a_G1_1535877165274, PREFETCH_COUNT: 300, LINK_CREDIT: 0, PREFETCH_Q_LEN: 0]
2018-09-02 08:32:45,500 WARN  [pool-2-thread-9] c.m.a.e.PartitionPump - host salvador-to-cassandra-758ed9cf-722d-4674-b58d-a68803f15738: 8: EventHub client error continued
java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:cd407ddfef2847d0a54fafa43b6c42bc_B13, SystemTracker:SharedConnectionListener, Timestamp:9/2/2018 8:32:45 AM TrackingId:2a1dd72a1af240b295ecaa259ca8928a_G1, SystemTracker:gateway5, Timestamp:9/2/2018 8:32:45 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/8, REFERENCE_ID: cf942a_928a_G1_1535877165274, PREFETCH_COUNT: 300, LINK_CREDIT: 0, PREFETCH_Q_LEN: 0]
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:86)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.drainPendingReceives(MessageReceiver.java:404)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:390)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:621)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:50)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:70)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:36)
	at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
	at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
	at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:483)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.eventhubs.EventHubException: No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:cd407ddfef2847d0a54fafa43b6c42bc_B13, SystemTracker:SharedConnectionListener, Timestamp:9/2/2018 8:32:45 AM TrackingId:2a1dd72a1af240b295ecaa259ca8928a_G1, SystemTracker:gateway5, Timestamp:9/2/2018 8:32:45 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/8, REFERENCE_ID: cf942a_928a_G1_1535877165274, PREFETCH_COUNT: 300, LINK_CREDIT: 0, PREFETCH_Q_LEN: 0]
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:58)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:620)
	... 15 more
2018-09-02 08:32:45,503 INFO  [pool-2-thread-5] c.m.a.e.i.ReceivePump - Stopping receive pump for partition (8) as pump ran into errors.
@royyeah royyeah mentioned this issue Sep 3, 2018
Closed
@SreeramGarlapati
Copy link
Contributor

SreeramGarlapati commented Sep 3, 2018 •
edited
Loading

That's right #373 - fixes this issue.
May I request you to try latest version of EPH with latest version (1.1.0) of JavaClient and ack here that they are compatible and helps with this issue..

@SreeramGarlapati SreeramGarlapati self-assigned this Sep 3, 2018
@royyeah
Copy link
Author

royyeah commented Sep 4, 2018 •
edited
Loading

Great! I have changed to 1.1.0, however since I have no idea when partitions are being moved across nodes at Azure, which makes it hard to verify if it solved the issue. Do you have an idea how we can verify this?

@SreeramGarlapati
Copy link
Contributor

@royyeah - if you enable warn level logs for EventHub client package (along with EPH) - you would see these errors in the EventHub client logs No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. and if you dont see corresponding partition move warn logs in EPH logs - there it is!

@royyeah
Copy link
Author

royyeah commented Sep 5, 2018

@SreeramGarlapati - thanks, I will keep an eye on it.

@SreeramGarlapati
Copy link
Contributor

Thanks a lot for acking that these two packages go well with each other. I was able to recommend to couple other customers based on that!
Am closing this as the code fix for this issue is in. Please feel free to reopen - if you see this again - @royyeah .

@royyeah
Copy link
Author

royyeah commented Sep 18, 2018 •
edited
Loading

@SreeramGarlapati

We encountered a similar error on two partitions and from the logs / data we are missing its seems both partitions never recovered (Java client 1.1.0 and EPH 2.0.1).

I am not able to reopen the issue, apparently if you are not a collaborator on GitHub you can only reopen issues you closed yourself. Can you reopen the issue?

Logs regarding error on partition 12:

2018-09-12 17:56:51,482 WARN  [pool-2-thread-7] c.m.a.e.i.ReceivePump - Receive pump for partition (12) exiting after receive exception java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: connection aborted, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/12, REFERENCE_ID: 3936cc_5d1_G16_1536744858067, PREFETCH_COUNT: 300, LINK_CREDIT: 259, PREFETCH_Q_LEN: 0]
2018-09-12 17:56:51,482 WARN  [pool-2-thread-7] c.m.a.e.PartitionPump - host salvador-to-cassandra-82412a74-5b1b-4e45-b97d-9b7b3b96ecee: 12: EventHub client error: java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: connection aborted, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/12, REFERENCE_ID: 3936cc_5d1_G16_1536744858067, PREFETCH_COUNT: 300, LINK_CREDIT: 259, PREFETCH_Q_LEN: 0]
2018-09-12 17:56:51,482 WARN  [pool-2-thread-7] c.m.a.e.PartitionPump - host salvador-to-cassandra-82412a74-5b1b-4e45-b97d-9b7b3b96ecee: 12: EventHub client error continued
java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: connection aborted, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/12, REFERENCE_ID: 3936cc_5d1_G16_1536744858067, PREFETCH_COUNT: 300, LINK_CREDIT: 259, PREFETCH_Q_LEN: 0]
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.drainPendingReceives(MessageReceiver.java:439)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:425)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:658)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:50)
	at com.microsoft.azure.eventhubs.impl.MessagingFactory.onConnectionError(MessagingFactory.java:286)
	at com.microsoft.azure.eventhubs.impl.ConnectionHandler.onTransportError(ConnectionHandler.java:124)
	at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191)
	at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
	at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:487)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.eventhubs.EventHubException: connection aborted, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/12, REFERENCE_ID: 3936cc_5d1_G16_1536744858067, PREFETCH_COUNT: 300, LINK_CREDIT: 259, PREFETCH_Q_LEN: 0]
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:60)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:657)
	... 15 more
2018-09-12 17:56:51,497 INFO  [pool-2-thread-9] c.m.a.e.i.ReceivePump - Stopping receive pump for partition (12) as pump ran into errors.

Logs regarding error on partition 10:

2018-09-13 09:40:25,397 WARN  [pool-2-thread-13] c.m.a.e.i.ReceivePump - Receive pump for partition (10) exiting after receive exception java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: com.microsoft.azure.eventhubs.impl.AmqpException: The link 'G22:101819893:4c5f0d_ade_G22_1536775026250' is force detached by the broker due to errors occurred in consumer(link1316600). Detach origin: InnerMessageReceiver was faulted. TrackingId:b8ba55bb00014b0f001416f85b995372_G22_B1, SystemTracker:enxnsdatalakealphaprd:eventhub:ehsalvador7readings~11263|cgcassandra, Timestamp:9/13/2018 9:40:25 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/10, REFERENCE_ID: 4c5f0d_ade_G22_1536775026250, PREFETCH_COUNT: 300, LINK_CREDIT: 247, PREFETCH_Q_LEN: 0]
2018-09-13 09:40:25,397 WARN  [pool-2-thread-13] c.m.a.e.PartitionPump - host salvador-to-cassandra-82412a74-5b1b-4e45-b97d-9b7b3b96ecee: 10: EventHub client error: java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: com.microsoft.azure.eventhubs.impl.AmqpException: The link 'G22:101819893:4c5f0d_ade_G22_1536775026250' is force detached by the broker due to errors occurred in consumer(link1316600). Detach origin: InnerMessageReceiver was faulted. TrackingId:b8ba55bb00014b0f001416f85b995372_G22_B1, SystemTracker:enxnsdatalakealphaprd:eventhub:ehsalvador7readings~11263|cgcassandra, Timestamp:9/13/2018 9:40:25 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/10, REFERENCE_ID: 4c5f0d_ade_G22_1536775026250, PREFETCH_COUNT: 300, LINK_CREDIT: 247, PREFETCH_Q_LEN: 0]
2018-09-13 09:40:25,397 WARN  [pool-2-thread-13] c.m.a.e.PartitionPump - host salvador-to-cassandra-82412a74-5b1b-4e45-b97d-9b7b3b96ecee: 10: EventHub client error continued
java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.EventHubException: com.microsoft.azure.eventhubs.impl.AmqpException: The link 'G22:101819893:4c5f0d_ade_G22_1536775026250' is force detached by the broker due to errors occurred in consumer(link1316600). Detach origin: InnerMessageReceiver was faulted. TrackingId:b8ba55bb00014b0f001416f85b995372_G22_B1, SystemTracker:enxnsdatalakealphaprd:eventhub:ehsalvador7readings~11263|cgcassandra, Timestamp:9/13/2018 9:40:25 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/10, REFERENCE_ID: 4c5f0d_ade_G22_1536775026250, PREFETCH_COUNT: 300, LINK_CREDIT: 247, PREFETCH_Q_LEN: 0]
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.drainPendingReceives(MessageReceiver.java:439)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:425)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:658)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:50)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:70)
	at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:36)
	at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
	at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
	at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:487)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.microsoft.azure.eventhubs.EventHubException: com.microsoft.azure.eventhubs.impl.AmqpException: The link 'G22:101819893:4c5f0d_ade_G22_1536775026250' is force detached by the broker due to errors occurred in consumer(link1316600). Detach origin: InnerMessageReceiver was faulted. TrackingId:b8ba55bb00014b0f001416f85b995372_G22_B1, SystemTracker:enxnsdatalakealphaprd:eventhub:ehsalvador7readings~11263|cgcassandra, Timestamp:9/13/2018 9:40:25 AM, errorContext[NS: enxnsdatalakealphaprd.servicebus.windows.net, PATH: ehsalvador7readings/ConsumerGroups/cgcassandra/Partitions/10, REFERENCE_ID: 4c5f0d_ade_G22_1536775026250, PREFETCH_COUNT: 300, LINK_CREDIT: 247, PREFETCH_Q_LEN: 0]
	at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:55)
	at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:657)
	... 15 more
Caused by: com.microsoft.azure.eventhubs.impl.AmqpException: The link 'G22:101819893:4c5f0d_ade_G22_1536775026250' is force detached by the broker due to errors occurred in consumer(link1316600). Detach origin: InnerMessageReceiver was faulted. TrackingId:b8ba55bb00014b0f001416f85b995372_G22_B1, SystemTracker:enxnsdatalakealphaprd:eventhub:ehsalvador7readings~11263|cgcassandra, Timestamp:9/13/2018 9:40:25 AM
	... 17 more
2018-09-13 09:40:25,402 INFO  [pool-2-thread-2] c.m.a.e.i.ReceivePump - Stopping receive pump for partition (10) as pump ran into errors.

Logs containing No connection handler was found for virtual host:

2018-09-11 13:21:20,149 INFO  [pool-2-thread-2] c.m.a.e.i.BaseLinkHandler - linkName[f52733_b3f7_G7_1536672079699], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G7'. TrackingId:dfc14de278ea440e867c9de59b9bcfea_B30, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:21:19 PM TrackingId:25ea29c3446e422ba83d7d79347eb3f7_G7, SystemTracker:gateway5, Timestamp:9/11/2018 1:21:19 PM]
2018-09-11 13:21:20,312 INFO  [pool-2-thread-15] c.m.a.e.i.BaseLinkHandler - linkName[b4a0f5_eec_G13_1536672079748], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G13'. TrackingId:dfc14de278ea440e867c9de59b9bcfea_B30, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:21:20 PM TrackingId:9c1a9611787b465789da5c3c7917beec_G13, SystemTracker:gateway5, Timestamp:9/11/2018 1:21:20 PM]
2018-09-11 13:44:14,024 INFO  [pool-2-thread-4] c.m.a.e.i.BaseLinkHandler - linkName[5f52d8_b07_G26_1536673453776], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G26'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:13 PM TrackingId:816bc74803f84fac80de3af0af303b07_G26, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:13 PM]
2018-09-11 13:44:15,002 INFO  [pool-2-thread-15] c.m.a.e.i.BaseLinkHandler - linkName[7cef16_0ed_G18_1536673454812], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:6bd94c4d1a1e4e35833be4209177f0ed_G18, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,013 INFO  [pool-2-thread-8] c.m.a.e.i.BaseLinkHandler - linkName[14cb6f_aac_G25_1536673454809], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:79f6f02496ef48f09d417b707233aaac_G25, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,042 INFO  [pool-2-thread-15] c.m.a.e.i.BaseLinkHandler - linkName[2fc183_eec_G13_1536673454806], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G13'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:9c1a9611787b465789da5c3c7917beec_G13, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,062 INFO  [pool-2-thread-15] c.m.a.e.i.BaseLinkHandler - linkName[bfada4_b3f7_G7_1536673454845], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G7'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:25ea29c3446e422ba83d7d79347eb3f7_G7, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,063 INFO  [pool-2-thread-4] c.m.a.e.i.BaseLinkHandler - linkName[b5a761_b3f7_G7_1536673454839], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G7'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:25ea29c3446e422ba83d7d79347eb3f7_G7, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,096 INFO  [pool-2-thread-8] c.m.a.e.i.BaseLinkHandler - linkName[bdc479_75c_G17_1536673454909], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G17'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:cd09ee753f4f4c0da2b967d51bd0175c_G17, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,119 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[82864a_0ed_G18_1536673454839], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:14 PM TrackingId:6bd94c4d1a1e4e35833be4209177f0ed_G18, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:14 PM]
2018-09-11 13:44:15,354 INFO  [pool-2-thread-9] c.m.a.e.i.BaseLinkHandler - linkName[a8a9f5_b07_G26_1536673455071], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G26'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:15 PM TrackingId:816bc74803f84fac80de3af0af303b07_G26, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:15 PM]
2018-09-11 13:44:16,752 INFO  [pool-2-thread-6] c.m.a.e.i.BaseLinkHandler - linkName[342964_0ed_G18_1536673456539], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:16 PM TrackingId:6bd94c4d1a1e4e35833be4209177f0ed_G18, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:16 PM]
2018-09-11 13:44:16,783 INFO  [pool-2-thread-16] c.m.a.e.i.BaseLinkHandler - linkName[f8c719_aac_G25_1536673456543], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:16 PM TrackingId:79f6f02496ef48f09d417b707233aaac_G25, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:16 PM]
2018-09-11 13:44:16,802 INFO  [pool-2-thread-9] c.m.a.e.i.BaseLinkHandler - linkName[f102f9_eec_G13_1536673456567], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G13'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:16 PM TrackingId:9c1a9611787b465789da5c3c7917beec_G13, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:16 PM]
2018-09-11 13:44:16,843 INFO  [pool-2-thread-9] c.m.a.e.i.BaseLinkHandler - linkName[d63112_b3f7_G7_1536673456591], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G7'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:16 PM TrackingId:25ea29c3446e422ba83d7d79347eb3f7_G7, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:16 PM]
2018-09-11 13:44:16,843 INFO  [pool-2-thread-14] c.m.a.e.i.BaseLinkHandler - linkName[2dba36_b3f7_G7_1536673456591], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19215'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G7'. TrackingId:651eb5a9eb004c95b482f781a4cb6603_B24, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 1:44:16 PM TrackingId:25ea29c3446e422ba83d7d79347eb3f7_G7, SystemTracker:gateway5, Timestamp:9/11/2018 1:44:16 PM]
2018-09-11 15:17:32,851 INFO  [pool-2-thread-12] c.m.a.e.i.BaseLinkHandler - linkName[dd9394_b965_G3_1536679051532], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G3'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:03e656e7b57543899a2c26f25d6ab965_G3, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:32 PM]
2018-09-11 15:17:32,871 INFO  [pool-2-thread-2] c.m.a.e.i.BaseLinkHandler - linkName[1ae87a_b965_G3_1536679051533], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G3'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:03e656e7b57543899a2c26f25d6ab965_G3, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:32 PM]
2018-09-11 15:17:32,891 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[8191c1_3fe_G18_1536679051483], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:5df1d49cf25a4bc89384769ee5d833fe_G18, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:32 PM]
2018-09-11 15:17:32,892 INFO  [pool-2-thread-7] c.m.a.e.i.BaseLinkHandler - linkName[609416_95c_G25_1536679051507], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:600eb3718da1463994c108ead6f0a95c_G25, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:32 PM]
2018-09-11 15:17:32,972 INFO  [pool-2-thread-16] c.m.a.e.i.BaseLinkHandler - linkName[8851aa_67b_G14_1536679051512], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G14'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:48f4f261ab1546e3819b6c112df0567b_G14, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:32 PM]
2018-09-11 15:17:33,317 INFO  [pool-2-thread-6] c.m.a.e.i.BaseLinkHandler - linkName[72b9e2_0b95_G8_1536679051575], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G8'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:bee75fd3c4874a6e85a8084cb0b50b95_G8, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:33 PM]
2018-09-11 15:17:33,402 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[569534_e1e_G20_1536679051524], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G20'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:32 PM TrackingId:c338805d6180425e81e45f12cc494e1e_G20, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:33 PM]
2018-09-11 15:17:33,587 INFO  [pool-2-thread-9] c.m.a.e.i.BaseLinkHandler - linkName[e1ca51_eec_G13_1536679051516], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G13'. TrackingId:77ddcf41b4a1473b9dc13e1d11c66a0c_B6, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:17:33 PM TrackingId:9c1a9611787b465789da5c3c7917beec_G13, SystemTracker:gateway5, Timestamp:9/11/2018 3:17:33 PM]
2018-09-11 15:39:56,570 INFO  [pool-2-thread-13] c.m.a.e.i.BaseLinkHandler - linkName[8e3fab_eddd_G1_1536680395206], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:86497536f6944be190cd8ce0e6831cd5_B14, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:39:56 PM TrackingId:5a61ce9318b04d1b89bf5980538aeddd_G1, SystemTracker:gateway5, Timestamp:9/11/2018 3:39:56 PM]
2018-09-11 15:39:56,980 INFO  [pool-2-thread-1] c.m.a.e.i.BaseLinkHandler - linkName[df962d_5d1_G16_1536680395138], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G16'. TrackingId:86497536f6944be190cd8ce0e6831cd5_B14, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:39:56 PM TrackingId:8543a553cd474e0a8e2950dc7a2935d1_G16, SystemTracker:gateway5, Timestamp:9/11/2018 3:39:56 PM]
2018-09-11 15:39:57,360 INFO  [pool-2-thread-3] c.m.a.e.i.BaseLinkHandler - linkName[930403_8e2_G29_1536680395184], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G29'. TrackingId:86497536f6944be190cd8ce0e6831cd5_B14, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:39:56 PM TrackingId:c9e9fc8669784db595a426602b2d18e2_G29, SystemTracker:gateway5, Timestamp:9/11/2018 3:39:57 PM]
2018-09-11 15:39:57,395 INFO  [pool-2-thread-2] c.m.a.e.i.BaseLinkHandler - linkName[5dbd1d_49b_G15_1536680395485], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G15'. TrackingId:86497536f6944be190cd8ce0e6831cd5_B14, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:39:56 PM TrackingId:e233970a66ad477898850ae9577c549b_G15, SystemTracker:gateway5, Timestamp:9/11/2018 3:39:57 PM]
2018-09-11 15:39:57,694 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[b93f4d_b3f7_G7_1536680395453], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G7'. TrackingId:86497536f6944be190cd8ce0e6831cd5_B14, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:39:56 PM TrackingId:25ea29c3446e422ba83d7d79347eb3f7_G7, SystemTracker:gateway5, Timestamp:9/11/2018 3:39:57 PM]
2018-09-11 15:39:57,794 INFO  [pool-2-thread-12] c.m.a.e.i.BaseLinkHandler - linkName[01d716_181_G23_1536680395594], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G23'. TrackingId:86497536f6944be190cd8ce0e6831cd5_B14, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 3:39:56 PM TrackingId:469804d485384d54b332431887228181_G23, SystemTracker:gateway5, Timestamp:9/11/2018 3:39:57 PM]
2018-09-11 17:05:57,119 INFO  [pool-2-thread-4] c.m.a.e.i.BaseLinkHandler - linkName[bac2c4_5d1_G16_1536685556815], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G16'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:56 PM TrackingId:8543a553cd474e0a8e2950dc7a2935d1_G16, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:56 PM]
2018-09-11 17:05:58,008 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[0d5095_ade_G22_1536685557813], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G22'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:57 PM TrackingId:0395aca6a61f45b2a5ac57268b786ade_G22, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:57 PM]
2018-09-11 17:05:58,018 INFO  [pool-2-thread-6] c.m.a.e.i.BaseLinkHandler - linkName[cb23fc_49b_G15_1536685557826], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G15'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:57 PM TrackingId:e233970a66ad477898850ae9577c549b_G15, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:57 PM]
2018-09-11 17:05:58,032 INFO  [pool-2-thread-7] c.m.a.e.i.BaseLinkHandler - linkName[0855a0_8a7_G20_1536685557806], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G20'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:57 PM TrackingId:f168194f87b74d81a722c798c695b8a7_G20, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:57 PM]
2018-09-11 17:05:58,058 INFO  [pool-2-thread-6] c.m.a.e.i.BaseLinkHandler - linkName[a96f05_181_G23_1536685557823], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G23'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:57 PM TrackingId:469804d485384d54b332431887228181_G23, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:57 PM]
2018-09-11 17:05:58,079 INFO  [pool-2-thread-6] c.m.a.e.i.BaseLinkHandler - linkName[fca4cb_eddd_G1_1536685557843], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:57 PM TrackingId:5a61ce9318b04d1b89bf5980538aeddd_G1, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:57 PM]
2018-09-11 17:05:58,085 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[2ecb10_8e2_G29_1536685557812], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G29'. TrackingId:e2974b2e882b45ffb4ec10008e407397_B9, SystemTracker:SharedConnectionListener, Timestamp:9/11/2018 5:05:57 PM TrackingId:c9e9fc8669784db595a426602b2d18e2_G29, SystemTracker:gateway5, Timestamp:9/11/2018 5:05:57 PM]
2018-09-12 09:34:12,933 INFO  [pool-2-thread-12] c.m.a.e.i.BaseLinkHandler - linkName[fdc39f_8f4_G11_1536744852595], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G11'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:12 AM TrackingId:89386d67981241c4ab4e46fe3ac2d8f4_G11, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:12 AM]
2018-09-12 09:34:13,756 INFO  [pool-2-thread-4] c.m.a.e.i.BaseLinkHandler - linkName[15fbfc_97c0_G4_1536744853554], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G4'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:e38748316f5e45edadba33a334d797c0_G4, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:13,766 INFO  [pool-2-thread-7] c.m.a.e.i.BaseLinkHandler - linkName[1b8fc3_95c_G25_1536744853556], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:600eb3718da1463994c108ead6f0a95c_G25, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:13,775 INFO  [pool-2-thread-3] c.m.a.e.i.BaseLinkHandler - linkName[4ab2e5_8c40_G0_1536744853555], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G0'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:8d55e104b3fc4541b6a75b1438ee8c40_G0, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:13,782 INFO  [pool-2-thread-9] c.m.a.e.i.BaseLinkHandler - linkName[fc22b2_ab5_G24_1536744853621], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G24'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:56ab308d5aab485bb267047222903ab5_G24, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:13,805 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[e4ece1_3fe_G18_1536744853554], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:5df1d49cf25a4bc89384769ee5d833fe_G18, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:13,809 INFO  [pool-2-thread-1] c.m.a.e.i.BaseLinkHandler - linkName[ef67bb_7b4_G17_1536744853556], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G17'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:31a3ab1d2e224cfb83d3de11e16b17b4_G17, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:14,028 INFO  [pool-2-thread-11] c.m.a.e.i.BaseLinkHandler - linkName[12f6fe_5d1_G16_1536744853654], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G16'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:13 AM TrackingId:8543a553cd474e0a8e2950dc7a2935d1_G16, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:13 AM]
2018-09-12 09:34:14,279 INFO  [pool-2-thread-16] c.m.a.e.i.BaseLinkHandler - linkName[567012_8f4_G11_1536744854027], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G11'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:14 AM TrackingId:89386d67981241c4ab4e46fe3ac2d8f4_G11, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:14 AM]
2018-09-12 09:34:15,567 INFO  [pool-2-thread-7] c.m.a.e.i.BaseLinkHandler - linkName[52e7bf_7b4_G17_1536744855367], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G17'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:31a3ab1d2e224cfb83d3de11e16b17b4_G17, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:15,570 INFO  [pool-2-thread-13] c.m.a.e.i.BaseLinkHandler - linkName[e9e60f_95c_G25_1536744855306], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:600eb3718da1463994c108ead6f0a95c_G25, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:15,572 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[dc88e0_97c0_G4_1536744855306], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G4'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:e38748316f5e45edadba33a334d797c0_G4, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:15,593 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[937a40_ab5_G24_1536744855326], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G24'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:56ab308d5aab485bb267047222903ab5_G24, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:15,601 INFO  [pool-2-thread-8] c.m.a.e.i.BaseLinkHandler - linkName[47df57_8c40_G0_1536744855325], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G0'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:8d55e104b3fc4541b6a75b1438ee8c40_G0, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:15,613 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[6a3273_3fe_G18_1536744855405], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:5df1d49cf25a4bc89384769ee5d833fe_G18, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:15,828 INFO  [pool-2-thread-14] c.m.a.e.i.BaseLinkHandler - linkName[de94d4_5d1_G16_1536744855567], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G16'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:8543a553cd474e0a8e2950dc7a2935d1_G16, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 09:34:16,103 INFO  [pool-2-thread-12] c.m.a.e.i.BaseLinkHandler - linkName[9d57cb_8f4_G11_1536744855828], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G11'. TrackingId:04ed6bfdb28e4812ba82873d5f4eb7e1_B16, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 9:34:15 AM TrackingId:89386d67981241c4ab4e46fe3ac2d8f4_G11, SystemTracker:gateway5, Timestamp:9/12/2018 9:34:15 AM]
2018-09-12 17:36:10,658 INFO  [pool-2-thread-7] c.m.a.e.i.BaseLinkHandler - linkName[fd5c49_b965_G3_1536773770385], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G3'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:10 PM TrackingId:03e656e7b57543899a2c26f25d6ab965_G3, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:10 PM]
2018-09-12 17:36:11,608 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[e2fb65_95c_G25_1536773771403], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:600eb3718da1463994c108ead6f0a95c_G25, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-12 17:36:11,653 INFO  [pool-2-thread-8] c.m.a.e.i.BaseLinkHandler - linkName[14e0fb_b965_G3_1536773771432], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G3'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:03e656e7b57543899a2c26f25d6ab965_G3, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-12 17:36:11,672 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[5abecf_0b95_G8_1536773771432], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G8'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:bee75fd3c4874a6e85a8084cb0b50b95_G8, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-12 17:36:11,689 INFO  [pool-2-thread-1] c.m.a.e.i.BaseLinkHandler - linkName[d24893_3fe_G18_1536773771425], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:5df1d49cf25a4bc89384769ee5d833fe_G18, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-12 17:36:11,691 INFO  [pool-2-thread-9] c.m.a.e.i.BaseLinkHandler - linkName[405606_8c40_G0_1536773771431], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G0'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:8d55e104b3fc4541b6a75b1438ee8c40_G0, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-12 17:36:11,699 INFO  [pool-2-thread-10] c.m.a.e.i.BaseLinkHandler - linkName[b551a8_eddd_G1_1536773771411], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G1'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:5a61ce9318b04d1b89bf5980538aeddd_G1, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-12 17:36:11,704 INFO  [pool-2-thread-16] c.m.a.e.i.BaseLinkHandler - linkName[ed48b6_49b_G15_1536773771391], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G15'. TrackingId:f55687c1a4fb4f6bbd0fa79bb44564d2_B30, SystemTracker:SharedConnectionListener, Timestamp:9/12/2018 5:36:11 PM TrackingId:e233970a66ad477898850ae9577c549b_G15, SystemTracker:gateway5, Timestamp:9/12/2018 5:36:11 PM]
2018-09-13 14:36:22,225 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[f9c5d3_b5f0_G6_1536849382019], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G6'. TrackingId:f1a2325e1df949e5a9014edcaf479887_B31, SystemTracker:SharedConnectionListener, Timestamp:9/13/2018 2:36:22 PM TrackingId:5c192757fc9e4ae6b02421f11353b5f0_G6, SystemTracker:gateway5, Timestamp:9/13/2018 2:36:22 PM]
2018-09-13 14:36:23,059 INFO  [pool-2-thread-13] c.m.a.e.i.BaseLinkHandler - linkName[8a55c1_66d_G13_1536849382810], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G13'. TrackingId:7bc8af72de5e48c3aacfa5dd40266b27_B27, SystemTracker:SharedConnectionListener, Timestamp:9/13/2018 2:36:22 PM TrackingId:48f8b2d0e282426381a9b6ad942c466d_G13, SystemTracker:gateway5, Timestamp:9/13/2018 2:36:22 PM]
2018-09-13 14:36:23,065 INFO  [pool-2-thread-1] c.m.a.e.i.BaseLinkHandler - linkName[cf25bc_95c_G25_1536849382836], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G25'. TrackingId:7bc8af72de5e48c3aacfa5dd40266b27_B27, SystemTracker:SharedConnectionListener, Timestamp:9/13/2018 2:36:22 PM TrackingId:600eb3718da1463994c108ead6f0a95c_G25, SystemTracker:gateway5, Timestamp:9/13/2018 2:36:22 PM]
2018-09-13 14:36:23,082 INFO  [pool-2-thread-2] c.m.a.e.i.BaseLinkHandler - linkName[e89cec_3fe_G18_1536849382821], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:7bc8af72de5e48c3aacfa5dd40266b27_B27, SystemTracker:SharedConnectionListener, Timestamp:9/13/2018 2:36:22 PM TrackingId:5df1d49cf25a4bc89384769ee5d833fe_G18, SystemTracker:gateway5, Timestamp:9/13/2018 2:36:22 PM]
2018-09-13 14:36:23,116 INFO  [pool-2-thread-16] c.m.a.e.i.BaseLinkHandler - linkName[71e902_8de4_G5_1536849382843], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G5'. TrackingId:7bc8af72de5e48c3aacfa5dd40266b27_B27, SystemTracker:SharedConnectionListener, Timestamp:9/13/2018 2:36:22 PM TrackingId:f4c81f107fc94f69a2f38eda50018de4_G5, SystemTracker:gateway5, Timestamp:9/13/2018 2:36:22 PM]
2018-09-13 14:36:23,116 INFO  [pool-2-thread-16] c.m.a.e.i.BaseLinkHandler - linkName[e7c0a5_97c0_G4_1536849382863], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19214'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G4'. TrackingId:7bc8af72de5e48c3aacfa5dd40266b27_B27, SystemTracker:SharedConnectionListener, Timestamp:9/13/2018 2:36:22 PM TrackingId:e38748316f5e45edadba33a334d797c0_G4, SystemTracker:gateway5, Timestamp:9/13/2018 2:36:22 PM]
2018-09-15 07:36:52,851 INFO  [pool-2-thread-13] c.m.a.e.i.BaseLinkHandler - linkName[209ea7_3fe_G18_1536997012578], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19216'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G18'. TrackingId:c1951c0499db4488beb3e74c152313d6_B18, SystemTracker:SharedConnectionListener, Timestamp:9/15/2018 7:36:52 AM TrackingId:5df1d49cf25a4bc89384769ee5d833fe_G18, SystemTracker:gateway5, Timestamp:9/15/2018 7:36:52 AM]
2018-09-17 04:37:23,766 INFO  [pool-2-thread-7] c.m.a.e.i.BaseLinkHandler - linkName[4a934c_8a7_G20_1537159043509], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G20'. TrackingId:e11184d1f45f4eeaada2fe45fd5645ef_B15, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 4:37:23 AM TrackingId:f168194f87b74d81a722c798c695b8a7_G20, SystemTracker:gateway5, Timestamp:9/17/2018 4:37:23 AM]
2018-09-17 23:37:31,584 INFO  [pool-2-thread-14] c.m.a.e.i.BaseLinkHandler - linkName[c77053_b9e_G28_1537227451143], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G28'. TrackingId:70070e9216cf47aa9b5b29f40435f504_B2, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 11:37:31 PM TrackingId:83dee3ab86bb4f8e8f1d62cb8ca60b9e_G28, SystemTracker:gateway5, Timestamp:9/17/2018 11:37:31 PM]
2018-09-17 23:37:32,205 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[f08c72_ab5_G24_1537227452034], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G24'. TrackingId:70070e9216cf47aa9b5b29f40435f504_B2, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 11:37:32 PM TrackingId:56ab308d5aab485bb267047222903ab5_G24, SystemTracker:gateway5, Timestamp:9/17/2018 11:37:32 PM]
2018-09-17 23:37:32,265 INFO  [pool-2-thread-5] c.m.a.e.i.BaseLinkHandler - linkName[4b06da_49b_G15_1537227452027], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G15'. TrackingId:70070e9216cf47aa9b5b29f40435f504_B2, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 11:37:32 PM TrackingId:e233970a66ad477898850ae9577c549b_G15, SystemTracker:gateway5, Timestamp:9/17/2018 11:37:32 PM]
2018-09-17 23:37:32,289 INFO  [pool-2-thread-6] c.m.a.e.i.BaseLinkHandler - linkName[3e7df9_b5f0_G6_1537227452014], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G6'. TrackingId:70070e9216cf47aa9b5b29f40435f504_B2, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 11:37:32 PM TrackingId:5c192757fc9e4ae6b02421f11353b5f0_G6, SystemTracker:gateway5, Timestamp:9/17/2018 11:37:32 PM]
2018-09-17 23:37:32,293 INFO  [pool-2-thread-15] c.m.a.e.i.BaseLinkHandler - linkName[3214fe_850_G14_1537227452026], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G14'. TrackingId:70070e9216cf47aa9b5b29f40435f504_B2, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 11:37:32 PM TrackingId:8fc0f096dae441ea922c7cc4d7ed8850_G14, SystemTracker:gateway5, Timestamp:9/17/2018 11:37:32 PM]
2018-09-17 23:37:32,338 INFO  [pool-2-thread-2] c.m.a.e.i.BaseLinkHandler - linkName[11285a_97c0_G4_1537227452030], ErrorCondition[com.microsoft:container-close, No connection handler was found for virtual host '19213'. Remote container id is '7A2E0A0515A9427F82D1ACF7F25A024D_G4'. TrackingId:70070e9216cf47aa9b5b29f40435f504_B2, SystemTracker:SharedConnectionListener, Timestamp:9/17/2018 11:37:32 PM TrackingId:e38748316f5e45edadba33a334d797c0_G4, SystemTracker:gateway5, Timestamp:9/17/2018 11:37:32 PM]

Please let me know if you need more info (full logs for example).

@royyeah royyeah mentioned this issue Sep 18, 2018
Closed
@sjkwak sjkwak reopened this Sep 19, 2018
@SreeramGarlapati
Copy link
Contributor

@royyeah - thanks a lot for reporting the issue and presenting the logs.
Earlier, I got carried away by the error message & parts of the stack trace & didnot realize a nuance - in the error stack trace (I guess I was under the influence of Confirmation bias :)).
#373 - implements retry mechanism on createReceiver call - whereas, the issue reported here is on the PartitionReceiver.Receive API call. It would not have fixed this - due to similarities in the error message and parts of stacktrace - I did not realize that - appologies about that.
The actual issue here is that: PartitionReceiver.Receive API - should always return null - when there are Transient errors (i.e., when EventHubException.isTransient()` is true) - and should never throw transient exceptions to the end user. However, due to a bug - in some cases - this is not happenning. And as a result - it throws this exception on the pump. Pump doesn't expect that and quits.
Since, EventHubs partitions are generally highly available and rarely have the downtime of over a minute - this was hard to reproduce. This is fixed in #381 (included as part of 1.2.0 release).
https://github.com/Azure/azure-event-hubs-java/blame/1.2.0/azure-eventhubs/src/main/java/com/microsoft/azure/eventhubs/impl/MessageReceiver.java#L433
Please go ahead and upgrade and enable warn level logs and open a new issue like you did earlier - if you see it again...

@royyeah
Copy link
Author

royyeah commented Sep 23, 2018

@SreeramGarlapati - great, thanks, I will try the new version and let you know if I see it again.

@yarinm yarinm mentioned this issue Oct 22, 2018
Closed
@christianor
Copy link

shouldn't the eph also be released with the newer dependency if this fixes this bug? i am also experiencing the same behaviour

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SreeramGarlapati SreeramGarlapati

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@christianor @royyeah @sjkwak @SreeramGarlapati