Add a new delayed subscription strategy for topics with retained mess…

[mths1]

Add a new delayed subscription strategy for topics with retained messages. Due to the fact that the retrain service and the message push are separate subsystems this is mitigate the unfortunate situation when a subscriber subscribes slightly after the messaging system has send the message, and slightly before the retrain sync has completed. This only affects clustered setups which use retrained messages.

Proposed Changes

• Add a new strategy that let subscribers wait for the retrain subystem to sync
• New messages will still be accepted, just delayed until the subsystem has synced
• After the sync, the retrained message is added to the beginning of the queue

Implications:

• The new function QFun in vmq_queue might hurt performance slightly - did not measure yet. Could maybe rewritten in a way that it uses pattern matching...
• It is possible to get the very same message twice (the retrained one, and the original one, based on some unfortunate timing)

This PR is just to see if there is interest in finalizing this, or if we want to stick with the original design.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes issue Puzzle: Sometimes Retained messages seem to be skipped #1633 )
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation (correction or otherwise)
• Cosmetics (whitespace, styles...)
• DevOps (Build scripts, pipelines...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask on the
mailing list. We're here to help! This is simply a reminder of what we are
going to look for before merging your code.

• I have read the CODE_OF_CONDUCT.md document
• All tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if needed)
• Any dependent changes have been merged and published in related repositories
• I have updated changelog (At the bottom of the release version)
• I have squashed all my commits into one before merging

Further Comments

If this is a relatively large or complex change, kick off the discussion by
explaining why you chose the solution you did and what alternatives you
considered, etc.

[rafaelnobrekz]

Thanks @mths1 for the proposal. Is there an easy way to test your PR on a server using VerneMQ?
@ioolkos Is there interest in adopting this kind of solution to the main code? Or is there another solution being discussed elsewhere? In our use case, there is initially no retained messages, so when we subscribe close to the latest publish, we get no messages at all and have to manually resubscribe after some threshold to get the published retained message.

[ioolkos]

@rafaelnobrekz you can test this PR by:

git fetch origin pull/2219/head:2219
git checkout 2219

You then have to compile this branch.
This PR is not a full solution but might mitigate the issue. Thank you for testing!