** (stop) :unexpected_delivery_and_no_default_consumer

[rafaeljesus]

Hi mates,

I am having a strange behavior running my consumer, once it is started by the main app supervisor, I see the following errors:

23:51:10.144 [error] GenServer #PID<0.328.0> terminating
** (stop) exited in: :gen_server2.call(#PID<0.327.0>, {:consumer_call, {:"basic.cancel_ok", "amq.ctag-acZaaX2r1zshRRALqkfieg"}, {:"basic.cancel", "amq.ctag-acZaaX2r1zshRRALqkfieg", false}}, :infinity)
    ** (EXIT) :unexpected_delivery_and_no_default_consumer
    (rabbit_common) src/gen_server2.erl:340: :gen_server2.call/3
    (amqp_client) src/amqp_channel.erl:745: :amqp_channel.handle_method_from_server1/3
    (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

It only happens on my Rabbit QA instance, locally it goes well,

The consumer do receives messages normally 🤔

Any thoughts?

[aeliton]

I have asked on the rabbitmq-users list and got the following answer:
https://groups.google.com/forum/#!topic/rabbitmq-users/VgtTpl4Yb14

I'm not sure my implementation is correct and I couldn't create a simple example program to reproduce this error (I'm getting that error intermittently in my phoenix application).

My latest try was to downgrade esl-elrlang to 19.3.6 and rabbitmq-server to 3.6.10, but I'm not convinced the problem won't happen again.

[ono]

Thanks @aeliton for sharing. Seeing number of 👍 , there are quite few people suffering the issue. I haven't managed to reproduce the issue. Any helps and information are welcome.

[ono]

Is it possible to get more detailed logs? For example...

def publish do
  {:ok, connection} = AMQP.Connection.open
  {:ok, channel} = AMQP.Channel.open(connection)
  publish(channel, "hello")
  ...
end

def publish(channel, queue)
  AMQP.Queue.declare(channel, queue)
  AMQP.Basic.publish(channel, "", queue, "Hello World!")
catch
  :exit, reason ->
    Logger.error "EXIT signal with #{reason} - fail to publish a message to #{queue}"
    status = AMQP.Queue.status(channel, queue) # < it fails if the queue doesn't exist.
    Logger.error "Queue status: #{inspect status}"

    exit(:rabbit_mq_error)
end

[archseer]

We had this happening during shutdown, if our terminate/2 didn't run and the channel never closed on time. Here's some related data: jcabotc/hare#8 (comment)

[zoltanmaric]

I've ran into this issue when using RabbitMQ as an MQTT broker and using an incorrect password. The client would connect successfully, but when attempting to subscribe to an MQTT topic, I would get the following error on the server:

2018-11-09 10:02:48.428 module=gen_server pid=<0.1559.0> [error]: GenServer #PID<0.1559.0> terminating
** (stop) exited in: :gen_server2.call(#PID<0.1558.0>, {:consumer_call, {:"basic.cancel_ok", "amq.ctag-qN5UkmBhOtnu0xYRxB3LJA"}, {:"basic.cancel"
    ** (EXIT) :unexpected_delivery_and_no_default_consumer
    (rabbit_common) /root/vpp/deps/rabbit_common/src/gen_server2.erl:329: :gen_server2.call/3
    (amqp_client) /root/vpp/deps/amqp_client/src/amqp_channel.erl:754: :amqp_channel.handle_method_from_server1/3
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:method, {:"basic.cancel_ok", "amq.ctag-qN5UkmBhOtnu0xYRxB3LJA"}, :none, :noflow}}

[ono]

There is a high chance the issue is fixed on 1.2.0-rc.0 so please give it a try. I changed the way the broker process monitors other process. I believe this is caused when the client pid is shut down before channel is closed.

[ono]

If you can make sure always closing a channel before the owner process gets down, that might help too.

We will release 1.2 shortly so please give it try. If neither solution - upgrading 1.2 or closing a channel - work, feel free to reopen the issue.

[karlseguin]

We're seeing something which I think is related. Our consumers essentially shut down on rescue/catch:

rescue  /
  _ ->
    Basic.reject(channel, message.delivery_tag, requeue: requeue)
    Basic.cancel(channel, consumer_tag)
    AMQP.Channel.close(channel)
   {:stop, normal, nil}

And we end up with "ghost" channels that have un-acked messages on the rabbitmq side. The simplest way to reproduce this issue reliably:

1 - Run the Consumer from this project's readme (the one that sets up gen_server_test_queue)
2 - Add the following to AMQP.SelectiveConsumer before the existing deliver_to_consumer_or_die/3:

defp deliver_to_consumer_or_die(_, {:basic_deliver, _, _}, _) do
  exit(:unexpected_delivery_and_no_default_consumer)
end

3 - Send a message to the queue

This is obviously extreme, but it's meant to show what happens when the SelectiveConsumer exits. If you explore the RabbitMQ interface, you should see a pending "ack". If you wait consumer_timeout amount of time (a) rabbitmq will start to log errors about "consumer ack timed out on channel 1" and you'll see in the interface, this "ghost" channel that has an unacked message but no consumers.

I would think that closing the channel would solve this issue, but it doesn't appear to. This very rarely happens, so I'm thinking it's some timing between the message being received, the :DOWN being received and the Channel close being processed on the RabbitMQ side.