Channels Not Closing in 2.0

[jbarket]

I have an application using 2.0 that's making calls between a number of microservices. Everything works as expected, no obvious errors, but channels in Rabbit are never disposed. If I start the app and make a request, I can see the number of channels in Rabbit tick up for every RequestAsync call I make.

Any general ideas on what I could be doing to cause RawRabbit to not close the channels once the request has been made?

[garethrampton]

I've seen this with 2.0 with both the RPC model (Request/Respond) and also message sequences. Standard pub/sub messaging appears to work fine and close/reuse channels.

In fact during testing pushing high volumes of messages via sequences actually results in the Rabbit refusing to respond (it effectively appears to shut down connections as auto deleting exchanges/queues disappear and appears to be unable to recover). Not sure whether this is a RabbitMQ client issue or with RawRabbit though.

[pardahlman]

Hello @jbarket and @garethrampton - thanks for reporting and commenting on this!

The original assumption for 2.0 was that the building of the middleware pipes would be expensive, which was why the resulting pipes was cached. This could have unwanted consequences, as it would cache a middleware with a dependency that was registered as transient.

This bug was introduced (or exposed if you want) when the cached pipe was removed, as the the IConsumerFactory was registered as transient and it held a instance specific list of consumers. There is a fix for this in the code base, namely making the list of consumer static. At the moment I'm blocked by #211 to publish this.

There are three thing you could try to do:

• Copy the implementation of ConsumerFactory and register in RawRabbits internal IoC
• You could try to register the existing ConsumerFactory as Singleton. I haven't tried this myself but I believe that it would resolve the problem. Your registration will override the default one
• Wait for next release, which is currently depending on the issue over at the NuGet repo

Hope this helps!

[garethrampton]

@pardahlman Tried suggestions 1 and 2 and it has made no difference in my case. I'm still seeing channels grow and never getting closed.

I'm also seeing RPC and message sequence failures with higher volumes (10/second) of messages - in the case of RPC the entire client connection appears to drop, and in the case of message sequences they simply fail and timeout. I can't but help thinking this is related to the number of open channels every increasing.

[pardahlman]

@garethrampton that's strange. I verified that the issue was not present in the 2.0 branch. I was able to reproduce it simply by making the list of consumers non-static, and then fix it by registering the IConsumerFactory as a Singleton. I committed the changes in a3db134.

It'd be interesting to know why you don't get the same result.

[pardahlman]

As for the message sequence.... There seem to be a some sort of limitation when performing multiple concurrent requests. I've been running the test for concurrent calls and gradually changing the number of concurrent calls to 100, 200, 300, 400... finally at 700 it failed with the following stack trace

System.TimeoutException : The operation has timed out.
   at RabbitMQ.Util.BlockingCell.GetValue(TimeSpan timeout)
   at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)
   at RabbitMQ.Client.Impl.ModelBase.BasicConsume(String queue, Boolean noAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IBasicConsumer consumer)
   at RabbitMQ.Client.Impl.AutorecoveringModel.BasicConsume(String queue, Boolean noAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IBasicConsumer consumer)
   at RawRabbit.Consumer.ConsumerFactory.ConfigureConsume(IBasicConsumer consumer, ConsumeConfiguration cfg)

The way I read this, there is a timeout that occures when trying to configure the consumer. The stack trace indicates that the TimeoutException is thrown in the underlying client, which possibly/probably is due to the number of channels and concurrent number of calls to the broker.

[pardahlman]

@jbarket @garethrampton - these fixes for this issue has been released in beta6. I also addressed the issue with Message Sequence (above).

[garethrampton]

@pardahlman Can confirm this as being fully fixed with the RPC pattern in beta6. No idea why I was still seeing this pre-beta6 despite implementing the fix. I can only assume I'd missed something!

[pardahlman]

Thanks for the update, @garethrampton - I'll be closing this now!