When Rabbit goes down a dicom is sent and comes back up, MIG cannot reconnect

[RemakingEden]

Description

I have been testing resiliency, one of the tests is to temporarily disconnect RabbitMQ. The behaviour while it is disconnected is correct but when i bring Rabbit back up. MIG proceeds to not be able to connect to it.

Steps to reproduce

Please share a clear and concise description of the problem.

1. Have MIG/MWM and all required containers running
2. Diconnect RabbitMQ
3. Try to send a dicom through the system
4. Reconnect RabbitMQ
5. Try to send a dicom through the system

Expected behavior

Dicom would go through workflow as before Rabbit is disconnected

Actual behavior

MIG complains it cannot connect to RabbitMQ and payload is deleted

Other information

Weirdly this seems to only happen if a dicom is attempted to be sent whilst Rabbit is offline.

2022-09-29T15:39:52.436062946Z <4> 15:39:52 Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationAct
ionHandler[706] => Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf =
> Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Payload=faa61cb
d-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf Failed to publish workflow request
for payload faa61cbd-5a5c-4a7a-9c6b-578a2be600ac; added back to queue for retry. RabbitMQ.Client.Exceptions.BrokerUnre
achableException: None of the specified endpoints were reachable  ---> System.AggregateException: One or more errors o
ccurred. (Connection failed)  ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed  ---> System.
Net.Sockets.SocketException (111): Connection refused    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.Th
rowException(SocketError error, CancellationToken cancellationToken)    at System.Net.Sockets.Socket.AwaitableSocketAs
yncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)    at System.Threading.Tasks.Value
Task.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) --- End of stack trace from previous location ---    at Ra
bbitMQ.Client.Impl.TcpClientAdapter.ConnectAsync(String host, Int32 port)    at RabbitMQ.Client.Impl.TaskExtensions.Ti
meoutAfter(Task task, TimeSpan timeout)    at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket,
 AmqpTcpEndpoint endpoint, TimeSpan timeout)    --- End of inner exception stack trace ---    at RabbitMQ.Client.Impl.
SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)    at RabbitMQ.Client.
Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout, Ad
dressFamily family)    at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingIPv4(AmqpTcpEndpoint endpoint, Func`2 so
cketFactory, TimeSpan timeout)    at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 so
cketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)    at RabbitMQ.Client.Framing.Im
pl.IProtocolExtensions.CreateFrameHandler(IProtocol protocol, AmqpTcpEndpoint endpoint, ArrayPool`1 pool, Func`2 socke
tFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)    at RabbitMQ.Client.ConnectionFac
tory.CreateFrameHandler(AmqpTcpEndpoint endpoint)    at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpo
intResolver resolver, Func`2 selector)    --- End of inner exception stack trace ---    at RabbitMQ.Client.EndpointRes
olverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)    at RabbitMQ.Client.Framing.Impl.Autorecov
eringConnection.Init(IEndpointResolver endpoints)    at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointRe
solver endpointResolver, String clientProvidedName)    --- End of inner exception stack trace ---    at RabbitMQ.Clien
t.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)    at RabbitMQ.Cli
ent.ConnectionFactory.CreateConnection(String clientProvidedName)    at RabbitMQ.Client.ConnectionFactory.CreateConnec
tion()    at Monai.Deploy.Messaging.RabbitMQ.RabbitMQConnectionFactory.<>c__DisplayClass6_0.<CreatConnection>b__1()
 at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode) --- End of stack trace from previous location ---    at System
.Lazy`1.CreateValue()    at Monai.Deploy.Messaging.RabbitMQ.RabbitMQConnectionFactory.<>c__DisplayClass5_0.<CreateChan
nel>b__1(String updateKey, Lazy`1 updateConnection)    at System.Collections.Concurrent.ConcurrentDictionary`2.AddOrUp
date(TKey key, Func`2 addValueFactory, Func`3 updateValueFactory)    at Monai.Deploy.Messaging.RabbitMQ.RabbitMQConnec
tionFactory.CreateChannel(String hostName, String username, String password, String virtualHost, String useSSL, String
 portNumber)    at Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService.Publish(String topic, Message messa
ge)    at Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler.NotifyPayloadReady(Payl
oad payload) in /app/src/InformaticsGateway/Services/Connectors/PayloadNotificationActionHandler.cs:line 137    at Mon
ai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationActionHandler.NotifyAsync(Payload payload, ActionB
lock`1 notificationQueue, CancellationToken cancellationToken) in /app/src/InformaticsGateway/Services/Connectors/Payl
oadNotificationActionHandler.cs:line 76
2022-09-29T15:39:52.441135463Z <6> 15:39:52 Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationAct
ionHandler[711] => Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf =
> Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Payload=faa61cb
d-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf Publishing workflow request message
 ID=f754afd5-08fd-4828-bff8-bcec6f07010f...
2022-09-29T15:39:52.442214223Z <6> 15:39:52 Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService[10000] =>
Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Payload=faa61cbd-
5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Payload=faa61cbd-5a5c-4a7a-9c6b-57
8a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Message ID=f754afd5-08fd-4828-bff8-bcec6f07010f. Ap
plication ID=16988a78-87b5-4168-a5c3-2cfc2bab8e54. Publishing message to rabbitmq-monai/monaideploy. Exchange=monaidep
loy, Routing Key=md.workflow.request.
2022-09-29T15:39:52.442280732Z <3> 15:39:52 Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationAct
ionHandler[728] => Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf =
> Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Payload=faa61cb
d-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf Reached maximum number of retries f
or notifying payload faa61cbd-5a5c-4a7a-9c6b-578a2be600ac ready, giving up.
2022-09-29T15:39:52.474908911Z <6> 15:39:52 Monai.Deploy.InformaticsGateway.Services.Connectors.PayloadNotificationAct
ionHandler[720] => Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf =
> Payload=faa61cbd-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf => Payload=faa61cb
d-5a5c-4a7a-9c6b-578a2be600ac, Correlation ID=c9bf0c3b-e9d8-49ae-ad53-3346f436febf Payload faa61cbd-5a5c-4a7a-9c6b-578
a2be600ac deleted.

[mocsharp]

Fix made in Project-MONAI/monai-deploy-messaging#79

[github-actions]

🎉 This issue has been resolved in version 0.3.2 🎉

The release is available on:

• GitHub Release