MQTT 5.0 / message-expiry-interval / if exceeded, not all messages are deleted

[JSchy65]

mosquitto broker version 1.6.15. (Oracle Linux 8)

The MQTT 5.0 Specification Chapter 3.3.2.3.3 "Message Expiry Interval" requires:
"If the Message Expiry Interval has passed and the Server has not managed to start onward delivery to a matching subscriber, then it MUST delete the copy of the message for that subscriber."

Currently, it seems that mosquitto is not doing that.

Start mosquitto with the following configuration:
sys_interval 1
max_queued_messages 20000

Use subscribers to permanently check some values:
mosquitto_sub -p 1883 -t '$SYS/broker/clients/connected'
mosquitto_sub -p 1883 -t '$SYS/broker/clients/total'
mosquitto_sub -p 1883 -t '$SYS/broker/store/messages/count'

what leads to:

clients/connected 3
clients/total 3
messages/count 53

mosquitto_sub -p 1883 -q 1 -t test/# -c -i abc123 -d

clients/connected 4
clients/total 4
messages/count 53

stop client / ctrl-c / -> disconnect

clients/connected 3
clients/total 4
messages/count 53

mosquitto_pub -p 1883 -q 1 -t test/xxx -m "Testtext" -i def456 -D publish message-expiry-interval 10 -d --repeat 1000

clients/connected 3
clients/total 4
messages/count 1053

Pause > message-expiry-interval (10 s)

clients/connected 3
clients/total 4
messages/count 1053

mosquitto_sub -p 1883 -q 1 -t test/# -c -i abc123 -d

no messages delivered to subscriber -> as expected -> ok

clients/connected 4
clients/total 4
messages/count 1033 !!! unexpected, there are only 20 messages removed instead of all ???

stop client / ctrl-c / -> disconnect

clients/connected 3
clients/total 4
messages/count 1033

mosquitto_sub -p 1883 -q 1 -t test/# -c -i abc123 -d

clients/connected 4
clients/total 4
messages/count 1013 !!! again, exact 20 messages are removed

It's the same result with -q 2

If you retry the last steps, you will see, that there are always exact 20 messages removed.

My expectation here is, to fulfill the above requirement, that ALL messages should be removed there because there is no client waiting for them and the message-expiry-interval has passed.

Thank You very much, Regards
Joachim Schachermayer

[ralight]

Thank you for the report and clear description of how to reproduce the problem. This issue has already been addressed in more recent versions of the broker, and I only intend to make security related fixes to the 1.6.x line of releases. Are you able to upgrade to 2.0.14?

(sorry for the duplicate comment, but the same point applies)

[JSchy65]

I tested now with Mosquitto 2.0.14 on Ubuntu 20.04.3 LTS. There is the same behaviour as described above but instead by 20, the number of messages is decreased by 40.

On Oracle Linux 8 there is only mosquitto 1.6.15 available. But we check if we can upgrade to 2.0.14.

[ralight]

I also tested and did not see incorrect behaviour. Please bear in mind that the important behaviour is whether the messages are delivered to the client or not, whether mosquitto has removed them from its internal store is a different point altogether - so checking the values in $SYS may give you a false impression.

[JSchy65]

Yes, I agree with you: whether the messages are delivered to the client or not is the important point

Because of the thing with the $SYS/broker/store/messages/count, I did another test to check if the counter internaly works as expected. Unfortunately, I found the unexpected behaviour, that it is possible to (totaly ?) block the delivery of messages to a subscriber.

It seems, that it is related with this magic number of 40 that I mentioned in my comment above that makes the difference.

mosquitto version 2.0.14 on Oracle Linux 8

Start mosquitto with the following configuration:
sys_interval 1
max_queued_messages 300

Use subscribers to permanently check some values:
mosquitto_sub -p 1883 -t '$SYS/broker/clients/connected'
mosquitto_sub -p 1883 -t '$SYS/broker/clients/total'
mosquitto_sub -p 1883 -t '$SYS/broker/store/messages/count'

what leads to:

clients/connected 3
clients/total 3
messages/count 53

mosquitto_sub -p 1883 -q 1 -t test/# -c -i abc123 -d

clients/connected 4
clients/total 4
messages/count 53

stop client / ctrl-c / -> disconnect

clients/connected 3
clients/total 4
messages/count 53

mosquitto_pub -p 1883 -q 1 -t test/xxx -m "Testtext" -i def456 -D publish message-expiry-interval 10 -d --repeat 40

clients/connected 3
clients/total 4
messages/count 93

Pause > message-expiry-interval (> 10 s)

mosquitto_pub -p 1883 -q 1 -t test/xxx -m "Testtext" -i def456 -D publish message-expiry-interval 60 -d --repeat 100

clients/connected 3
clients/total 4
messages/count 193

Pause < message-expiry-interval ( < 60 s)

mosquitto_sub -p 1883 -q 1 -t test/# -c -i abc123

clients/connected 4
clients/total 4
messages/count 193

!!!!!!!! subscriber gets NO message !!!!!!!!!!

Same test but with --repeat 39 (instead 40) in the first mosquitto_pub
->
Subscriber gets 100 messages
works as expecxted

For me it seems, that if the number of messages the expiry interval is passed for, exceeds the 40 (I guess 20 in mosquitto 1.6.15), then the output of messages is blocked to that subscriber ?

[ralight]

I'm looking into this now. Just for your information though, you can also do this, which I personally find more convenient:

mosquitto_sub -p 1883 -v -t '$SYS/broker/clients/connected' -t '$SYS/broker/clients/total' -t '$SYS/broker/store/messages/count'

[JSchy65]

Thank you for the tip above. That makes things easier :-)

In meantime I have tested both test cases above with 2.0.15 and both are working as expected -> issue fixed.

Thank you very much for your support.