[ADDED] MQTT: test/bench using external client

[levb]

This PR adds a number of MQTT-specific benchmarks, executed with an external test client (see ConnectEverything/mqtt-test#1). This came out of my work on MQTT message delivery issues in 2.10, and is now packaged as a go test --bench, and added to CI.

The test client uses the Eclipse paho library, and can be executed standalone, pointed at an MQTT server. It measures the timing of its operations, and outputs it as JSON of reportable metrics.

The benchmarks here run the client, and report back the metrics it outputs. Memory allocations are measured server-only.

Github action benchmark is used to compare the benchmark results to the base ref of the PR, or the prior result in a branch. If it fails a 120% threshold, the failure is reported as a comment on the commit(!), and the build fails.

To measure the effect of parallel-fetching retained messages (SUBRET), added disable_retained_message_cache MQTT option to the server.

The following benchmarks are run, covering standalone/cluster, QOS, message sizes to 100K, as well as concurrent clients.

• PUB measures just publishing
• PUBRET measures publishing with the retained flag on
• PUBSUB measures message delivery times from publisher to subscriber
• SUBRET measures delivery of 100* retained messages to new subscriptions

Example of a failure (I rolled back the retained message "fanout" commit):

if you click on the comment link, you get:

TODO

• CON measure CONNECT to CONNACK, with stored sessions
• SUB measuring SUBSCRIBE to SUBACK

[derekcollison]

So this adds in an option to disable the retained msg cache. When would one want to do this? Why are we exposing it I guess is my question.

PR looks fine, just looking for some clarifications.

[levb]

@derekcollison the option is used only for testing, and is exposed because we use config files in the test utility functions to initialize servers/clusters.

I could move the flag into the (server.)srvMQTT struct, set it after the server(s) have been started, would that be preferable? If there's a better mechanism of passing "test options" into a server, please advise.

[derekcollison]

Let's see what @kozlovic has to say, he has done a few of these testing flags / switches for things.

[kozlovic]

Assuming the proposed change does not cause races (tests not running in parallel), it could be a better way since we would not introduce an non exposed option.

[kozlovic]

LGTM

[wallyqs]

LGTM