Question about persistence and how configure it properly

[svanharmelen]

I thought it was probably best to open an issue to ask for some guidance around how to properly configure (and understand) persistence. Let me make a short list so it's easy to track my questions:

1. I know that internally there is a queue that gets filled with commands and that a dedicated thread continuously pulls commands from this queue to process them. What I don't know is when a command is persisted? Is it persisted immediately when it's put onto queue and then removed again when it's successfully processed?

2. What role does maxBufferedMessages play when configuring persistence? Does it limit the maximum amount of commands that are allowed (at any given time) to be in the queue (so in memory) or does it limit the maximum amount of commands that will be persisted? Or (which I assume) are these two things actually tied to each other?

3. So just to double check my assumptions, is it correct that every command that is persisted is always in the queue as well? Either because it was put there while the client was running, or it was read back from persistence during creation of the client (assuming restoreMessages is set)?

4. Does deleteOldestMessages only delete the oldest commands from the in memory queue or also from the persistence store?

5. Can (or should) both sendWhileDisconnected and allowDisconnectedSendAtAnyTime be set to allow sends before the first connect? Of should sendWhileDisconnected be unset (0) and only allowDisconnectedSendAtAnyTime be set (1)?

6. What does maxInflight configure exactly? Does this limit the amount of commands that are allowed to be processed concurrently? So does this effect (for example) how many publish commands are being send to a server at the same time?

7. Are there any limitations when using the default persistence store? And/or is there any benefit in writing a user-defined persistence story over using the default one?

8. Looking at #1035 Increased persistence key buffer size #1036, does this also impact user-defined persistence stores (as it mentions PERSISTENCE_MAX_KEY_LENGTH which sounds pretty generic)?

9. Again looking at #1035 Increased persistence key buffer size #1036, does #define PERSISTENCE_SEQNO_LIMIT 1000000 mean that whatever persistence store you use, you can never persist more then 1000000 commands?

I get that these are quite some questions and most likely some of the question overlap a little, but I thought it would be good to have all these questions cleared up in a single issue (also for others who might search for similar insights).

Thanks!!

[svanharmelen]

I know it are a lot of questions, but it would be extremely helpful to get these insights and confirmations... So if anyone can spare 5 minutes I would be super grateful!

[icraggs]

I've been doing some other MQTT work this week (Sparkplug specification).

I thought it was probably best to open an issue to ask for some guidance around how to properly configure (and understand) persistence. Let me make a short list so it's easy to track my questions:

1. I know that internally there is a queue that gets filled with commands and that a dedicated thread continuously pulls commands from this queue to process them. What I don't know is _when_ a command is persisted? Is it persisted immediately when it's put onto queue and then removed again when it's successfully processed?

That's the idea, yes. To reduce or eliminate (hopefully) the windows between data persistence to protect against loss of data.

2. What role does `maxBufferedMessages` play when configuring persistence? Does it limit the maximum amount of commands that are allowed (at any given time) to be in the queue (so in memory) or does it limit the maximum amount of commands that will be persisted? Or (which I assume) are these two things actually tied to each other?

It limits the size of the command queue. I didn't intend at the start that the queue should be used to hold lots (tens or hundreds of thousands) of messages - better to use a database for that.

3. So just to double check my assumptions, is it correct that every command that is persisted is always in the queue as well? Either because it was put there while the client was running, or it was read back from persistence during creation of the client (assuming `restoreMessages` is set)?

The in memory and persisted queues should have the same content. See answer 4.

4. Does `deleteOldestMessages` only delete the oldest commands from the in memory queue or also from the persistence store?

Both. They should be kept in sync. The persistence store is only there to enable the queue to be recreated if the app is ended and started up again.

5. Can (or should) both `sendWhileDisconnected` and `allowDisconnectedSendAtAnyTime` be set to allow sends before the first connect? Of should `sendWhileDisconnected` be unset (0) and only `allowDisconnectedSendAtAnyTime` be set (1)?

sendWhileDisconnected doesn't allow sends before the first connect. This seemed like a good approach to me to avoid messages being queued for destinations which aren't valid. But I was asked to allow it, so allowDisconnectedSendAtAnyTime allow sends before the first connect. They both need to be set.

6. What does `maxInflight` configure exactly? Does this limit the amount of commands that are allowed to be processed concurrently? So does this effect (for example) how many publish commands are being send to a server at the same time?

The number of concurrent "inflight" messages for QoS 1 and 2. Inflight being unacked - that is PUBACK for QoS 1 or PUBCOMP for QoS 2 hasn't yet been received.

7. Are there any limitations when using the default persistence store? And/or is there any benefit in writing a user-defined persistence story over using the default one?

It wasn't intended to be highest performance or have fancy features - just a basic example.

8. Looking at #1036, does this also impact user-defined persistence stores (as it mentions `PERSISTENCE_MAX_KEY_LENGTH` which sounds pretty generic)?

It limits the key size that can be used for user defined persistence mechanisms too.

9. Again looking at #1036, does `#define PERSISTENCE_SEQNO_LIMIT 1000000` mean that whatever persistence store you use, you can never persist more then 1000000 commands?

Yes. I would reiterate that I don't think storing any where near that number of commands is a good idea. If you need that many, used some more robust/high peforming data store.

I get that these are quite some questions and most likely some of the question overlap a little, but I thought it would be good to have all these questions cleared up in a single issue (also for others who might search for similar insights).

Thanks!!

[svanharmelen]

Thank you sooooo much for these answers @icraggs They really help to improve and tweak our setup!

Yet it does trigger another few questions 😏

1. In the mean time we have created a database backed user persistence store, but if I understand things correctly the maximum amount of persisted messages will still be 1.000.000 (which I think is more then enough by the way)? And in order to achieve that number, we should also set maxBufferedMessages to 1.000.000?

2. I noticed that the default persistence store never had a filename with a number higher then 65535. That's well below the 1.000.000 max we configured. Can you explain that and confirm that (when using our custom store) it will actually store a maximum of 1.000.000 messages instead or 65535?

2. What role does maxBufferedMessages play when configuring persistence? Does it limit the maximum amount of commands that are allowed (at any given time) to be in the queue (so in memory) or does it limit the maximum amount of commands that will be persisted? Or (which I assume) are these two things actually tied to each other?

It limits the size of the command queue. I didn't intend at the start that the queue should be used to hold lots (tens or hundreds of thousands) of messages - better to use a database for that

3. So how could we use a database for that? Is that possible with this package?

9. Again looking at #1035 Increased persistence key buffer size #1036, does #define PERSISTENCE_SEQNO_LIMIT 1000000 mean that whatever persistence store you use, you can never persist more then 1000000 commands?

Yes. I would reiterate that I don't think storing any where near that number of commands is a good idea. If you need that many, used some more robust/high peforming data store.

4. So do you mean something like we've just build (a database backed persistence store) or are you talking about something else here?

[svanharmelen]

Hmm... Guess the answer to questions 3 and 4 of my last update could be to use the callbacks i.c.w. a dedicated persistence solution separate of this package. Was that what you were referring to as well? Or do you mean something else?

[icraggs]

You can use a database with this package by implementing a user defined persistence store, yes. But if you store say 500k messages, there will still be an overhead of indexing etc, which I've tried to make fairly efficient but it won't be as good as a dedicated key value store.

If you wanted to be able to queue up several million messages then better to do that in a completely separate database before even touching the MQTT client.

Thinking about it, you can queue up more than 64k QoS 0 messages. But only up to 64k QoS 1 plus QoS 2 (the total). This is because there are 64k MQTT packet ids (QoS 0 messages don't have them), and the packet id is assigned before adding to the queue (due to a user request several years ago). So you can queue up to a million messages, but only 64k of those can be QoS 1 or 2. I hadn't thought about it like that before.

[svanharmelen]

Got it. Thanks for the info and in that case we are going to have to change our setup a little.

So just to double check/confirm; if we set maxBufferedMessages to 1000 and persistence to ::MQTTCLIENT_PERSISTENCE_NONE it will allow 1000 messages to be cached/queued by the MQTT client, but only in memory and it will not use any of the persistence related code/logic?

And if you then try to add the 1001th message, I assume you get an error on that action without interrupting anything already cached/queued, right?

[icraggs]

Got it. Thanks for the info and in that case we are going to have to change our setup a little.

So just to double check/confirm; if we set maxBufferedMessages to 1000 and persistence to ::MQTTCLIENT_PERSISTENCE_NONE it will allow 1000 messages to be cached/queued by the MQTT client, but only in memory and it will not use any of the persistence related code/logic?

And if you then try to add the 1001th message, I assume you get an error on that action without interrupting anything already cached/queued, right?

Yes.

[svanharmelen]

Check… Thanks 👍🏻