Automatic clean up of retained messages with expired message expiry interval

[asx64]

Hi,

We are currently using an expiry interval for retained messages with different topics to make sure that unrequired messages get automatically deleted by the broker. We recently noticed that the metrics exposed by the topics "$SYS/broker/store/messages/count" and "$SYS/broker/retained messages/count" increased to unexpected high values although we set message expiry intervals. From tests / observations we conducted we conclude that expired retained messages are currently only removed if new subscriptions on their topics are created by clients.
In our usage scenario this behavior currently causes a constantly increasing retained message count and of course memory amount. Thus, as a workaround we consider to call mosquitto_sub -t '#' regularly to trigger the message clean up.

Is there a mosquitto setting, we missed, that triggers regularly an automatic clean up of expired retained messages?

Steps to reproduce our observations

1. starting mosquitto broker version 2.0.20 with default settings;
2. observation of metric "$SYS/broker/store/messages/count" with mosquitto_sub -t '$SYS/broker/store/messages/count';
3. initially metric reports 51 stored messages;
4. publishing 10 retained messages with an expiry interval of 10 seconds with following bash script:

for i in $(seq 1 10);
do
    mosquitto_pub -t "/test/$i" -r -m test -D publish message-expiry-interval 10 -V 5
done

5. the metric increases to 61 (as expected); nevertheless, stays at this value even after 10 seconds;
6. calling mosquitto_sub -t '/test/1' reduces the the message count to 60;
7. calling mosquitto_sub -t '#' decreases the count to 51 - the initial value;

Related code lines

The removal of expired retained messages seems to take place in the code at following lines:

mosquitto/src/retain.c

Lines 161 to 166 in 7f50849

	if(branch->retained->message_expiry_time > 0 && db.now_real_s >= branch->retained->message_expiry_time){
	db__msg_store_ref_dec(&branch->retained);
	branch->retained = NULL;
	#ifdef WITH_SYS_TREE
	db.retained_count--;
	#endif

[ralight]

I'm curious about your use case, can you provide any more details? Is the memory use particularly prohibitive?

[ralight]

I agree with your observations by the way.

[asx64]

Thanks for your response.

I will try to roughly explain our use case in a general way.
It should first be mentioned that in our usage scenario our programs not only use retained messages but they are causing the described issue.
We share properties of unique entities between multiple programs. Some of them add and update property values (value: retained message payload) for specific entities (topic e.g. entities/ID/property) as soon as the programs can retrieve them. Other programs, which can partly start / connect / subscribe later on, monitor the properties of specific entities (by ID) for some time. These programs can immediately retrieve already provided property values from the broker but also further updates to them.
The property values of the entities have to be kept for a while by the broker since we have no explicit indication when they can be deleted. Nevertheless, after e.g. a day, if the properties were no longer updated, they can be safely removed as these entities will never be seen by the programs again. With the retained messages' expiry interval we want to ensure that the data at some point always gets automatically removed, without having an extra program / logic regularly doing a clean up.

We don't have a low memory restriction here but since, during a runtime of days / weeks, thousands of unique entities are being observed by the programs the expired retained messages start to build up infinitely as we currently do not subscribe to topics with expired messages / all topics.

I hope I could explain our use case.

[ralight]

Thank you, I understand. What you're doing is a bit unusual but certainly valid.

[ralight]

There will be a fix in 2.0.21 where you can set the retain_expiry_interval option - note that it's off by default.