Timeout::Error Timeout::Error

[blanchma]

I'm having an issue in one of the environments in which I run Bunny. It's weird because it's only happen in one env. The error is a Timeout. I don't share the connection between multiple threads but I have a Singleton to only build one session to prevent every time I sent a message a create a new session. If I build a session per message, the timeout dissapears. I'll like to understand what is going on in the line 33:

          if @q.empty?
            @cond.wait(@lock, timeout)
            raise ::Timeout::Error if @q.empty?
          end
          item = @q.shift
          @cond.signal

          item
        end

Stacktrace

/opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/concurrent/continuation_queue.rb:33:in `block in poll' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/concurrent/continuation_queue.rb:30:in `synchronize' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/concurrent/continuation_queue.rb:30:in `poll' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/session.rb:1261:in `wait_on_continuations' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/session.rb:503:in `block in open_channel' /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/session.rb:496:in `open_channel' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/channel.rb:223:in `open' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/session.rb:345:in `block in create_channel' /usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize' /opt/rubyapp/concourse/releases/20170505184556/vendor/bundle/ruby/2.3.0/gems/bunny-2.6.2/lib/bunny/session.rb:340:in `create_channel'

[michaelklishin]

Please post question to the mailing list in the future.

An operation that expects a response times out. This is not an indication of an issue in the client but it happens that you are running a version that does not include #462 (and I only know this from the stack trace, please do report versions of the libraries used when seeking help).

[michaelklishin]

See server logs for any possible alarms and that both the app's and node's TCP buffers aren't too small.

[blanchma]

I updated to 2.6.5 and the error continues.

[michaelklishin]

@blanchma the stack trace says a channel.open times out. See server logs for clues. I cannot suggest anything else with the amount of information provided.

[blanchma]

I still have this issue with 2.7.0 and still clueless about the issue:

W, [2017-08-14T19:38:12.395107 #20749]  WARN -- #<Bunny::Session:0x3ab89b8 hvgearln@sharp-cheetah.rmq.cloudamqp.com:5672, vhost=hvgearln, addresses=[sharp-cheetah.rmq.cloudamqp.com:5672]>: Recovering from connection.close (COMMAND_INVALID - second 'channel.open' seen)
W, [2017-08-14T19:38:12.395365 #20749]  WARN -- #<Bunny::Session:0x3ab89b8 hvgearln@sharp-cheetah.rmq.cloudamqp.com:5672, vhost=hvgearln, addresses=[sharp-cheetah.rmq.cloudamqp.com:5672]>: Will recover from a network failure (no retry limit)...

My connection class is like this:

class AmqpConnection
  include Singleton
  attr_reader :connection

  def initialize
    puts "::Configuration.amqp_url: #{::Configuration.amqp_url}"
    #@connection = MarchHare.connect(uri: ::Configuration.amqp_url)
    connection
  end

  def reconnect
    puts "[AmqpConnection] New Connection"
    @connection = Bunny.new(Configuration.amqp_url).start
  end

  def self.config
    @config ||= begin
      uri = URI(Configuration.amqp_url)
      {host: uri.host,
      username: uri.user,
      password: uri.password,
      vhost: uri.path}
    end
  end

  def open?
    @connection && @connection.status != :closed
  end

  def connection
    if open?
      @connection
    else
      reconnect
    end
  end

  def self.create_channel
    instance.connection.create_channel(nil, 128)
  end
end

Any time I want to publish or suscribe, I use AmqpConnection.create_channel but I get this Timeout::Error with too much frecuency.

[michaelklishin]

The code does no synchronization and does not demonstrate how channels or connections are used.

Take a traffic capture with Wireshark and see what's actually sent on the wire.