[Feature Request]: Do not retain MQTT 'stat' topic

[hb9eue]

Platform

other

Description

I have been investigating once more, why my mosquitto MQTT broker, to which I connect my nodes and which I bridge to the official mqtt server, is periodically running out of memory.

It looks like the stat topic has retained flag set. So they accumulate over time. Not sure why this is done this way. Would it be possible to turn that off to save memory on mqtt brokers?

-Benoît-

[GUVWAF]

Did you confirm this really solves your memory problem? The official MQTT server is very active. Depending on your hardware, I'm not surprised you will be running out of memory if you bridge it.

You could try if changing true to false at those two places changes memory usage:
https://github.com/meshtastic/firmware/blame/master/src/mqtt/MQTT.cpp#L334
https://github.com/meshtastic/firmware/blame/master/src/mqtt/MQTT.cpp#L444

[hb9eue]

Not yet. I have disconnected my MQTT bridge for now. I realized that if I delete all 'retained' messages via my two way bridge this would be propagated to the main MQTT server and that would not be so nice I guess.

What I wonder. This issue first appeared maybe a month or two ago. Then I disconnected my bridge, cleaned out the whole msh tree, re-connected and it was fine again.

On 17. June, the issue was back. My MQTT broker runs on a small arm64 board.

What could be related is this error being logged every couple of seconds when I connect the bridge:

1719039336: Client local.pine64.Meshtastic disconnected due to malformed packet.
1719039342: Connecting bridge (step 1) Meshtastic (mqtt.meshtastic.org:1883)
1719039342: Connecting bridge (step 2) Meshtastic (mqtt.meshtastic.org:1883)

I'm not sure if my mosquitto instance is disconnecting because it receives a malformed packet from the meshtastic mqtt broker, or if the mestastic broker is disconnecting me.

[hb9eue]

1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/paho210741302260803', ... (7 bytes))
1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/paho542566730235072', ... (7 bytes))
1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/!8487e2a0', ... (7 bytes))
1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/paho543244763548087', ... (7 bytes))
1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/!1f9ff154', ... (6 bytes))
1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/paho263754787787922', ... (7 bytes))
1719051163: Received PUBLISH from local.pine64.Meshtastic (d0, q0, r1, m0, 'meshtastic/msh/EU_868/2/stat/paho468566236478057', ... (7 bytes))
1719051163: Client local.pine64.Meshtastic disconnected due to malformed packet.

I guess the bridge is receiving a malformed packet from mqtt.meshtastic.org and when publishing this to my local mosquitto it is being disconnected, repeating with each re-connection to mqtt.meshtastic.org when all retained messages are received and re-published.

[hb9eue]

It is indeed the PUBLISH to the topic right before the disconnect which causes the issue. Mostly they start with paho...

After reconnecting, it disconnects on the same paho... topic again.

When I then go on and delete that retained message on mqtt.meshtastic.org the disconnect happens on the next such paho... message.
Not all paho... messages seem to cause the issue. When I look at them with MQTT-Explorer, I don't see what's wrong with them.

[hb9eue]

Now it's weird...

tail -f /var/log/mosquitto/mosquitto.log | grep --line-buffered -B 1 "Meshtastic disconnected due to malformed packet" | grep --line-buffered -o "'.*'"

This is getting me the topic which causes the disconnect.

If i then do a mosquitto_pub to mqtt.meshtastic.org with -r -n to delete that topic I see the next one.

So I attempted to replicate and before deleting the topic, I did a mosquitto_sub on it to then pass it on to my local broker with mosquitto_pub. For sure not the same as when the broker does the same in bridge mode. But even when looking at the connection with wireshark, I don't see anything obvious.

I had a glimpse at the connecting from my bridge to mqtt.mosquitto.org but it looks like if I subscribe to msh/# I get all retained messages in one go.

[hb9eue]

eclipse/mosquitto#3073 indicates it's an issue with the mosquitto broker bridge and high volume traffic.