MQTT connection silently fails after 5 minutes

[IoTThinks]

Hi all,
I'm using ESP32 Dev board with Arduino-mqtt and WiFi.
And connecting to ThingsBoard MQTTs. The ThingsBoard MQTT is confirmed OK with external Windows MQTT client.

The MQTT session keeps failing after 5 minutes.
I set KeepAlive to 10 and 60. No effect.

Any help or advice is appreciated.
Thanks a lot.

[IoTThinks]

The publish command and subscribe commands stop working after 5 minutes too.
Anything wrong with the KeepAlive?

[IoTThinks]

Reducing the KeepAlive to below 5 seconds seems to fix the problem.
mqttClient.setOptions(2, false, 1000);

Don't know why.
Let me monitor it for a night and I will update back.

[IoTThinks]

Reducing the KeepAlive to 5 seconds and timeout to 10 secconds help a lot.
But I would like to have a counter to keep track failed Keep Alive.
We only close the connect after 3 consecutive failed Keep Alive.

Sometimes the server is just slow to response or the network is slow.

Proofs
In this code https://github.com/256dpi/arduino-mqtt/blob/master/src/lwmqtt/client.c
If the pong is coming but the old pong is pending, the timeout will be issued.

In this code https://github.com/256dpi/arduino-mqtt/blob/master/src/MQTTClient.cpp
If the PONG is timeout, then the session is disconnected immediately.

Suggestion
We should have a counter as parameter to allow up to xxx number of failed KeepAlive before closing connection.

[256dpi]

I have the feeling your application suffers from more severe underlying transport issues. The keep alive just brings the problems to the surface. To me it looks like an idle connection is not properly kept alive either by the server, client or hardware in between. Would you be able to describe your setup?

[IoTThinks]

Hi 265dpi,
Thanks a lot for your awesome MQTT library.
I have a ESP32 board with WiFi as MQTT client.
And I use ThingsBoard as MQTT broker in my VPS. Yes, it has a MQTT broker inside.

I have a pfSense firewall between MQTT client and MQTT broker.

1. To clear the doubt that the issue is in my ThingsBoard, I use MQTTfx as MQTT client. It's able to maintain the session for days. I can publish the data to MQTT server successfully anytime.
   

2. At the beginning, I use default keep alive (10s), clean_session (true) and timeout (1000).
   The session will fail to publish or get subscribed message around 5 minutes after that.
   I even measure the time with my clock timer.
   

3. So I suspect, the default Option is too short for a MQTT broker in data center.
   If I make the KeepAlive too long, the issue happens again. May be the firewall doesn't like long idle connection?
   So I make the KeepAlive to 5s and timeout to 15s.
   Then increase the RAM for my MQTT server. :)
   And it runs ok for a day now.
   

4. If in bad luck, ONE keepalive misses the server then the connection will be dropped by your library.
   Normally, in servers, we have heartbeats (similar to MQTT ping).
   Only after three consecutive failed heartbeat, the connection is dropped.
   This is to minimize the bad luck or intermittent network / server issues.

Thanks for reading a long thread.

[IoTThinks]

Fixed already.
I set the keepalive as above.

And let loop() has ENOUGH MCU cycles to run.
Last time, other tasks may sometimes take long MCU cycles before loop() has a chance to run.

To close as resolved.
Thanks everyone.

[256dpi]

Glad it worked out!

[ortegafernando]

Fixed already.
I set the keepalive as above.

And let loop() has ENOUGH MCU cycles to run.
Last time, other tasks may sometimes take long MCU cycles before loop() has a chance to run.

To close as resolved.
Thanks everyone.

What is your final configuration then? Thanks

I have a configuration with SIM800L, Arduino Pro Mini and TinyGSM and I have the same problem (I think that it is the same problem).

I have an sketch with pubsubclient or this great library. With pubsubcliente everything goes well

And with this library it couldn't connect. After putting setPptions(10, true, 15000) it has improve a lot, but I am disconnected sometimes.
May be is the keep alive time, but I want to understand this and know your final configuration.

Thanks.

[IoTThinks]

Fixed already.
I set the keepalive as above.
And let loop() has ENOUGH MCU cycles to run.
Last time, other tasks may sometimes take long MCU cycles before loop() has a chance to run.
To close as resolved.
Thanks everyone.

What is your final configuration then? Thanks

I have a configuration with SIM800L, Arduino Pro Mini and TinyGSM and I have the same problem (I think that it is the same problem).

I have an sketch with pubsubclient or this great library. With pubsubcliente everything goes well

And with this library it couldn't connect. After putting setPptions(10, true, 15000) it has improve a lot, but I am disconnected sometimes.
May be is the keep alive time, but I want to understand this and know your final configuration.

Thanks.

I'm sorry to say.
Despite all configuration, the MQTT connection is still up and down.
So I changed to PubSubClient and never have MQTT disconnection any more to MQTT inside my ThingsBoard.
May due do ThingsBoard MQTT. But as long as they work together, then ok for me.

[ortegafernando]

Check this: #188 (just in case)