out of memory crash + disconnection in exit of context manager when using qos=2 (paho 1.3.1)

[axd1967]

for some dark reason, an out of memory error + disconnection occurs when publishing a qos=2 message.

I've created a small Vagrant VM that runs a RabbitMQ config and a client that attempts to send such a message: https://github.com/axd1967/paho-bug

the bug seems to happen upon exit of a context manager in publish() in paho/mqtt/client.py(1121):

paho.mqtt.python/src/paho/mqtt/client.py

Line 1121 in e9914a7

return message.info

(note: code simplified, I can successfully send qos=0 and qos=1 messages via RabbitMQ)

paho-mqtt (1.3.1) , Ubuntu 16

[axd1967]

• hmmm... just noticed - could this be due to RabitMQ MQTT plugin not dealing with QoS=2?
  see top of https://www.rabbitmq.com/mqtt.html
• another possibility: Use mutexes as context managers #168

[axd1967]

Aha! qos=2 is a no-go for MQTT for specific reasons:

• https://groups.google.com/d/msg/rabbitmq-users/wwpFdPCRAD4/XMV5Ia2xCQAJ
• https://groups.google.com/forum/#!searchin/rabbitmq-users/mqtt$20qos$3D2|sort:date/rabbitmq-users/AVKOnNzX_7w/NEiYB6zICQAJ
• https://groups.google.com/d/msg/rabbitmq-users/ecLWQhHBg1g/N_ZIUwe6ojIJ
• https://groups.google.com/d/msg/rabbitmq-users/wwDNboPmNaU/1IIwvDBwe9cJ

I think that closes part of the issue (but the crash should not happen)

in fact, the MQTT plugin could maybe issue big fat warnings on any QoS=2 use encountered, rather than let users find out? (but it is indeed documented on the top of the MQTT plugin doc page).

[PierreF]

Thank for your patience and report detailed report.

RabbitMQ don't like QoS=2 package, but on my quick test (docker run rabbitmq, then rabbitmq-plugins enable rabbitmq_mqtt) I don't reproduce an out of memory. The client only get disconnected as soon as it publish the message and reconnect (in loop). Which is expected if the broker don't support them :)

I will try to test you Vagrant file in the next days.

[axd1967]

indeed, that seems to be what is happening (disconnect + reconnect, but including an out-of-memory error)
(edit added: Ubuntu 16)

[PierreF]

Are you saying you have out of memory because the disconnect error is "Out of memory." ?
There is no out-of-memory in this cause, just a wrong number for this error :)

This come from

paho.mqtt.python/src/paho/mqtt/client.py

Line 1794 in e9914a7

return 1

which use "1" instead of an MQTT_ERR_XXX.

So except for a wrong error message, there is no other issue.

[axd1967]

did you try the Vagrant approach?

I crafted it specially, the README explains the steps to reproduce the issue.

why does a disconnection happens? it looks like qos=2 is not correctly dealt with.

[PierreF]

What I reproduce is RabbitMQ closing the connection (because it don't support QoS=2). On the client side I don't see any bad behavior (except the message / return code which is not the good one).

That why I asked if the "Out of memory" come only from the error_string(result). If that the case, the only issue on paho side is that a return 1 is used for a generic error, but MQTT_ERR_NOMEM also have the value 1. Paho should be updated to return a most useful error.

Paho can't do anything better with QoS=2 and RabbitMQ, the broker don't support it and you should either use only QoS < 2 or use another broker.

[YoungPyDawan]

At least issue from
#250 (comment)
is fixed in 1.6 branch:
https://github.com/eclipse/paho.mqtt.python/blame/1.6.x/src/paho/mqtt/client.py#L2373

by commit 332834e
so close?

[ralight]

I think so, yes, thanks for flagging this.