Support Rabbitmq´s MQTT plugin

[petersilva]

Sibling enhancement to #128 rather than have Sarracenia communicate at the application level to MQTT brokers, just take advantage of the mqtt plugin built-in to rabbitmq.
Could have an option like so:

rabbitmqtt_post_broker amqp://feeder@localhost
rabbitmqtt_post_exchange amq.topic

Given those settings, Sarracenia would initiate an AMQP connection to the given rabbitmq broker, and publish to the rabbitmqtt_post_exchange. the topic would be
prefixed the value of the post_exchange

So this should be easily done with a low-level tweak, perhaps in sr_message, or sr_amqp.

[petersilva]

Haven´t mentioned it yet, but obviously, the target rabbitmq broker must have mqtt plugin enabled and configured to match the settings above. Then something published on amqp as:

Exchange: xpublic,   topic:  20190616.GTS.CWAO.SA ...

must show up in the MQTT exchange as topic:   xpublic/20190616/GTS/CWAO/SA ...

[petersilva]

Once we can publish for MQTT subscribers, a little additional work would be needed to enable a foreign (non-sarracenia) MQTT client could publish to the hierarchy on the rabbitmq broker, and sarracenia could subscribe via amqp without modification...

I think all we need is:

rabbitmqtt_exchange amq.topic

and the sarracenia consumer, will know to take the first topic, and pretend that is an exchange, reversing the transformation above.

[petersilva]

so with three options, and some small tweaks in message parsing, we could get likely one model of mqtt interop working.

[petersilva]

Perhaps this is sufficiently confusing that we should write the documentation first? no... has to be written after because examples... can´t be written until the code exists... ok never mind.

[benlapETS]

Note1: All I have for now about this issue are questions :

• How are Sarra users flexible to learn new configurations ?
• Is it too much for the sole goal of supporting MQTT ?
• Will it be confusing even with good documentation ?

In brief, I would like to get some feedback from Sarra users (mostly analyst?) about their feeling on providing 2 solutions (#128 and #227) to one problem adding multiple new configurations ?

Note2: I want to provide a diagram of an overview of this current use case, I will work on this today.

[petersilva]

review of offline chat with @benlapETS :

• Sarracenia makes a point of not adding any abstraction of the underlying services. It tries to use the AMQP broker as-is, the apache web server as-is, ssh server as-is, so the configuration of those things are documented by the respective tools.

• The rabbitmq mqtt plugin is a very specific technology, and building an abstraction around it to hide the implementation is likely to be a lot more difficult to work with going forward. The plan outlined above is to have the skinniest of wrapping explicitly around the rabbitmq plugin, to allow sarracenia components to use it with minimal changes.

• the analysts we could consult with effectively are the two of us, as no-one else in our team has much exposure to mqtt brokers or the plugin for rabbitmq. It is very difficult to have productive consultations without working code.

• Probably the best way to consult with the analysts is to write the code and document it as EXPERIMENTAL. THEN we can have analysts kick the tires, and provide feedback. This method makes sense, because this feature is likely pretty short to implement. It's harder to do if the investment to get to basic function is much higher (as will likely be the case for use with non-rabbitmq brokers (interoperability) aka. MQTT #128)

• We could implement use with non-rabbitmq brokers (interoperability) aka. MQTT #128 the same way (tagged EXPERIMENTAL) which allows us to remove the functionality later if we conclude that it isn't a good path. I think we will end up keeping both though.

[benlapETS]

Note1: The right procedure would be to tag as EXPERIMENTAL in the .rst files and in the appropriate man pages I guess, isnt it ?

[petersilva]

yes, the man pages for sure. I guess the setup instructions would go in the Admin.rst, and the something should indicate the features are experimental.

[petersilva]

thinking about it... I don't think we even need the rabbitmqtt_post_broker setting. We just use the ordinary post_broker, the only setting we need is rabbitmqtt_post_exchange... when set the already described transformations happen to the topic and exchange, and we just use the existing post_broker.

[benlapETS]

Note1. Here are the detailed use cases for #227:

1. Sarracenia user activated RabbitMQ MQTT plugin (RabbitMQTT) support. Sarracenia should update the permissions that will enable the default RabbitMQTT user to publish, subscribe to topics and consume MQTT messages through the RabbitMQTT exchange. It would be preferable if RabbitMQTT queues comply with Sarracenia queues naming scheme, if not possible the new RabbitMQTT permissions should not compromise Sarracenia permission model in term of security and usability. With the new RabbitMQTT permissions, the Partner users is not allowed to publish messages and this should not required new configurations for a Sarracenia user.

2. Sarracenia user want to publish a message to a RabbitMQTT exchange which is available. He defines the destination ('post_') broker and the destination ('post_') exchange where to send messages. He also defines a destination ('post_') topic prefix where he specify the format version of the message and the AMQP exchange he would normally use at the end of the prefix (ie. v03.post.xpublic). Then, Sarracenia will read those configurations, connect to the broker and bind a queue to the exchange. Sarracenia will publish the message to the topic, usually a relative path name, preceded by the topic prefix.

3. Sarracenia user want to consume a message from a RabbitMQTT exchange which is available. He defines the source ( '' ) broker and the source ( '' ) exchange from where it consumes messages. He also defines a source ( '' ) topic prefix where he specify the format version of the message and the AMQP exchange he would normally subscribe at the end of the prefix (ie. v03.post.xpublic). Then, Sarracenia will read those configurations, connect to the broker and bind the queue to the RabbitMQTT exchange following the accept and reject patterns for the topics. Sarracenia will then consume messages it will receive from that subscription

4. A Partner User consume MQTT messages from a Sarracenia node that has MQTT enabled with an external MQTT client. He receive the messages as it flows through the RabbitMQTT exchange and its topic subscription.

Note2: Questions:

• Why would an analyst want to consume MQTT feed ?
  Nourredine told me that it is a possibility in an upgrade need from AMQP to MQTT standard but not in the near future.
• If the end user doesnt use Sarracenia he wont get the data, because those MQTT client won't download files for him as Sarracenia does. So, is it pointless to add this support or is it exactly what the end user wants, implement the download functionality itself ?
  To give users the ability (and the responsibility) to do anything they want with the data is exactly what we want and what some end users want.

[petersilva]

Note 2 questions... yes if all publishing is done from Sarracenia, and mqtt is only a consumer, then it doesn''t make much sense to consume from mqtt. I'm not entirely sure of the permission model. After we have done the first iteration, we may find different use cases, where we allow MQTT users to publish to the mqtt side of the broker, and we want to forward to others.

The other, much simpler/obvious use case is for loopback testing with rabbitmqtt_post_broker which would likely be a good thing to add as a flow_test. chain (add rabbitmqtt plugin on flow test, publish and then consume through it.)

We won't have a Sarracenia MQTT consumer until #128 is done, but if we were only worried about Sarracenia, then there is no need for mqtt in the first place. The purpose of adding MQTT is to demonstrate interop with other implementations... we could try it with the https://github.com/MetPX/wmo_mesh/ project as a sample consumer, but there are no existing implementations to interop with yet. so...

The motivations are:

Build it and they will come - https://www.youtube.com/watch?v=5Ay5GqJwHF8 (providing an intial reference implementation for others to work with.)

Dogfooding - https://medium.com/@benbob/the-gospel-of-dogfooding-can-i-hear-an-amen-brother-1af4d82cf221 ( using it ourselves to prove it works. )

[petersilva]

oh, yeah... there is another motivation for MQTT support... the dialup data acquisition (DARTS) stuff will evolve over the next few years, and there seems to be industry weight behind MQTT, so it is possible that the auto stations will produce MQTT data natively. Getting the data acquisition and exchange worlds closer together would be a good thing... but again, implementations have to exist to for people to work with.

[benlapETS]

Requirements

General:

R.i227 Sarracenia must support RabbitMQ´s MQTT plugin (aka RabbitMQTT)

Detailed (new or preserved):

Functionnal requirements (FR):

FR1 Sarracenia must provide a MQTT configuration interface that enables the uses of the RabbitMQ MQTT plugin.

• FR1.1 (new) Sarracenia must detect the activation state of RabbitMQTT.
• FR1.2 (new) Sarracenia must resolve RabbitMQTT exchange (to remove if using default).
• FR1.3 (new) Sarracenia must resolve RabbitMQTT user.
• FR1.4 (new) Sarracenia must provide a configuration that enables the consumption from RabbitMQTT.
• FR1.5 (preserved) Sarracenia must provide a configuration for the source ( '' ) broker.
• FR1.6 (preserved) Sarracenia must provide a configuration for the source ( '' ) exchange.
• FR1.7 (new) Sarracenia must provide a configuration that enables the publication to RabbitMQTT.
• FR1.8 (preserved) Sarracenia must provide a configuration for the destination ( 'post' ) broker.
• FR1.9 (preserved) Sarracenia must provide a configuration for the destination ( 'post' ) exchange.

FR2 Sarracenia must support MQTT through RabbitMQTT.

• FR2.1 (new) Sarracenia must be able to subscribe to RabbitMQTT with a topic.
• FR2.2 (new) Sarracenia must be able to consume a message received from a RabbitMQTT exchange.
• FR2.3 (new) Sarracenia must be able to publish a message to a RabbitMQTT exchange.
• FR2.4 (new) Sarracenia must be able to do the round tripping loopback AMQP->RabbitMQTT->AMQP
• FR2.5 (new) Sarracenia must provide a MQTT interface for Partner Users (to remove if all done by RabbitMQTT plugin).

FR3 Sarracenia must map RabbitMQTT to normal Sarracenia usage.

• FR3.1 (new) Sarracenia must encode the RabbitMQTT topic as a concatenation of the origin AMQP exchange name (the prefix) and the accept configuration (the suffix).
• FR3.2 (new) Sarracenia must decode the RabbitMQTT message to V02 messages (verify if it needs more features than the V03toV02 decoding).
• FR3.3 (new) Sarracenia must encode its V02 message to RabbitMQTT message. (verify if it needs more features than the V02toV03 encoding).

Non functionnal requirements (NR)

Reliability

• NR1 (preserved) Sarracenia must be compliant with V03 message.

Interoperability

• NR2 (new) Sarracenia must support MQTT permission model.

Security

• NR3 (preserved) Sarracenia must rely on AMQP permission model.
• NR4 (preserved) Sarracenia must allow anonymous users in read-only mode.

[benlapETS]

Note1: The actual configurations exchange and post_exchange are quite confusing, I propose to change it to: from_exchange and to_exchange. In fact, I would change ('';'post') semantics to ('from'; 'to') which would gives better readability for configurations.

[petersilva]

changing will break a lot... Could add an alias though...

[benlapETS]

Note 1: I don't intend to change syntax right now, I'm just taking notes from my thoughts. I am kind of building a list of future proposition of enhancement if we make a big version change. But if your ok to add the alias right away, I will do it...
Note 2: For better interoperability, I think it would be a good idea to use the default exchange which is amq.topic. It is usable as is when we activate the plugin. So without the need of specifying any exchange user should only have to activate RabbitMQTT and they will be ready to consume and/or publish on a remote broker more easily. My opinion is that only if we want to use amq.topic for two purposes we would have to decides which purpose will use it and which one will have to use another exchange. Also it will have least impact in sr_audit if we keep amq.topic for MQTT.

[benlapETS]

Note 1: My last thoughts on this issue will surely change some reasoning that has already been done on MQTT support with RabbitMQTT. Having already done some experimentations using the flow test, I realized I could create a flow of MQTT data by redirecting normal AMQP exchange flow to the designated MQTT exchange of one particular broker. To do that, I used this shovel configuration, without disrupting Sarracenia code:

post_broker amqp://tfeed@${FLOWBROKER}/
post_exchange rabbitmqtt
post_topic_prefix  v03.post.xflow_public

So, this config enabled RabbitMQTT publishing and MQTT consuming through a third-party user running mosquitto_sub. Then, to consume through Sarracenia I used this subscriber config:

broker amqp://tsub@${FLOWBROKER}/
exchange rabbitmqtt
topic_prefix v03.post.xflow_public

These config have already been integrated to the flow test, will need to document that and then discuss on the details, if this integration suites our needs for the RabbitMQ MQTT plugin support.

[benlapETS]

Note1: I wrote some howto doc to enable MQTT with Sarracenia, this is a first draft, but I tested it on my other machine and it worked. The only problem is that I had to authorized anonymous to have .* read permission (couldn't restrict to only x.*public).

[petersilva]

the name used in the howto, xmqtt_public exchange name doesn't match the required naming scheme ... which is why you needed to tweak permissions... the exchange name needs to start with xs_ and end in puiblic. Try an exchange name: xs_mqtt_public, and I imagine anonymous will be able to read it without the permission tweak. I think every sr_audit run will reset the permissions anyways... the recipe is likely very fragile as-is, it might even delete the exchange if run with the right options.

The permission model (which is based on standardized name scheme for exchanges) is described here:

https://github.com/MetPX/sarracenia/blob/master/doc/sr_subscribe.1.rst#naming-exchanges-and-queues

and a slightly older (and conflicting... ought to fix that ;-) here:

https://github.com/MetPX/sarracenia/blob/master/doc/Concepts.rst#security-considerations

[benlapETS]

Note1: The problem is not only the exchange scheme, in order to allow Partner user subscribers (like mosquitto_sub) to a feed we should allow a different queue scheme which I don't know how to control the format (for now) and which start with mqtt-subscription. Here is how the queue naming works in the RabbitMQ MQTT plugin https://github.com/rabbitmq/rabbitmq-mqtt/blob/master/src/rabbit_mqtt_util.erl:

subcription_queue_name(ClientId) ->
    Base = "mqtt-subscription-" ++ ClientId ++ "qos",
    {list_to_binary(Base ++ "0"), list_to_binary(Base ++ "1")}.

That means we would need ^mqtt-subscription in the configure and write regexp.

Note2: For the exchange name I just thought that as we were having xpublic and xflow_public for names that we use to implement and to test public exchanges, xmqtt_public would make sense for a public mqtt exchange name. And as it would be the only mqtt exchange of a node it would make sense to use it as standard for our mqtt nodes which would make it more interoperable and less confusing then having the name of a user in it... We would also protect it the same way as xpublic. Also, you proposed the xmqtt_public name yourself in #128

Note3: As I experimented sr_audit doesn't reset permission it only creates it, unless the reset flag is set to true which is not the case in the flow test, because we uses sr_audit --users foreground.

Note4: In the branch issue227, I've already changed the code on July12 (3dce17e) so xmqtt_public was protected as xpublic is... oh yeah, forgot that, it was rabbitmqtt but changed it to xmqtt_public yesterday. I also noticed that I should've committed the doc in the branch instead of master, sorry about that it could be a bit confusing.

Note5: Renamed to xs_mqtt_public (9d0e01f) we'll test with that until we discuss more about that.

[petersilva]

hmm... that is an issue... ok, if we need to change the permission mask, the solution is to modify sr_audit to enforce it always, and update documentation to suit. need to think about this for a while.

[petersilva]

ok... we discussed how you had proposed incorporating the topic prefix into the hierarchy at the end. This has the effect of putting all mqtt traffic into a single hierarchy. The reason for putting the exchange at the beginning of the tree was to be able to establish independent topic hierarchies (this things create data structures on mqtt brokers, much like exchanges on amqp ones.) so putting the exchange at the beginning spreads them out. I worry that piling everything into a single v03.post hierarchy will have adverse effects. Is this real? dunno, thinking....

[petersilva]

the other worry is that there isn't any way to identify a posting (by the content itself, without knowing) that has the exchange as the first post, versus one that doesn't.... should we perhaps differentiate using v03.postx. (if it has postx, then following subtopic is an exchange id?)

[petersilva]

released in 2.19.09