Flood publishing causes socket to be nil

[cviedmai]

There seems to be some kind of concurrency issues when I try to write a hell lot of messages locally. I've managed to reproduce the error by just running this code:

require 'bunny'
connection = Bunny.new({host: 'localhost', keepalive: true})

connection.start
channel = connection.create_channel
exchange = channel.topic("general", durable: true)

while true
  exchange.publish("hola", routing_key: 'resources', timestamp: Time.now.to_i, persistent: true)
end

after a while you will get this error:

undefined method `read_fully' for nil:NilClass
/Users/viki/.rvm/gems/ruby-1.9.3-p362@oceanus/gems/bunny-0.9.0.pre7/lib/bunny/transport.rb:112:in `read_next_frame'
/Users/viki/.rvm/gems/ruby-1.9.3-p362@oceanus/gems/bunny-0.9.0.pre7/lib/bunny/main_loop.rb:50:in `run_once'
/Users/viki/.rvm/gems/ruby-1.9.3-p362@oceanus/gems/bunny-0.9.0.pre7/lib/bunny/main_loop.rb:31:in `block in run_loop'
/Users/viki/.rvm/gems/ruby-1.9.3-p362@oceanus/gems/bunny-0.9.0.pre7/lib/bunny/main_loop.rb:28:in `loop'
/Users/viki/.rvm/gems/ruby-1.9.3-p362@oceanus/gems/bunny-0.9.0.pre7/lib/bunny/main_loop.rb:28:in `run_loop'
...

which comes continuously given that the Exception is swallowed in the main_loop

I've managed to avoid the error by setting single threaded true:

connection = Bunny.new({host: 'queue.dev.viki.io', keepalive: true, threaded: false})

[michaelklishin]

RabbitMQ will block publishers that overrun consumers. In theory it should not cause any socket exceptions but presumably it does. Network error handling is very simplistic right now, it is true.

[michaelklishin]

Can you count after about what # of messages published you get this behavior? And post what's in the RabbitMQ log? Thanks!

[michaelklishin]

Also, what's approximate median message size? So far I cannot reproduce this with small messages (1K) and 100,000 messages published.

[cviedmai]

It happens at about 270K messages. Some other details about my environment:

• I'm writing to a topic exchange.
• There is a durable queue that gets the messages from the exchange.

The message size doesn't really seems to matter. It happens even with 4B messages.
I believe the queue getting messages from the exchange is relevant.

These are the last lines in my Rabbit logs:

=INFO REPORT==== 15-Feb-2013::10:30:55 ===
closing AMQP connection <0.24366.1> (127.0.0.1:64089 -> 127.0.0.1:5672)

=INFO REPORT==== 15-Feb-2013::10:30:56 ===
accepting AMQP connection <0.24407.1> (127.0.0.1:64092 -> 127.0.0.1:5672)

=ERROR REPORT==== 15-Feb-2013::10:30:56 ===
closing AMQP connection <0.24407.1> (127.0.0.1:64092 -> 127.0.0.1:5672):
{bad_header,<<1,0,2,0,0,0,25,65>>}

[michaelklishin]

That means RabbitMQ somehow gets an empty frame header, considers it a fatal error and closes the connection.
Do you have an example script that reproduces the issue? What RabbitMQ version do you run?

[cviedmai]

Yes, the script I posted in the first message reproduces the issue when there is a durable queue subscribed to the exchange.
I'm running 3.0.1

[michaelklishin]

Can you check that RabbitMQ log does not have anything related to alarms when this happens?

I cannot reproduce this. My theory is that when your machine goes low on memory or disk space (< 1 GB), RabbitMQ blocks the publisher, eventually OS socket buffer gets full and this results in exceptions
that clear @socket in the transport. And because the publisher is not stopped, you see this nil pointer
exception.

[michaelklishin]

Setting RabbitMQ memory watermark to a really low value suggests it is not the issue:

sudo rabbitmqctl set_vm_memory_high_watermark 0.05

will cause the publisher quickly get blocked but it does not cause the script to fail with any exceptions.

[michaelklishin]

I can't reproduce it across 2 OSes, 2 RabbitMQ versions, 2 Ruby implementations. We have a test for a very
similar scenario, unless an example that reproduces it can be found, I consider this issue resolved.

[michaelklishin]

0.9.0.pre8 is out, please give it a try.