Support of MQTT

[didier-durand]

Is your feature request related to a way you would like Akri extended? Please describe.

MQTT is a key protocol in the IoT world. It would be extremely beneficial if Akri could directly communicate with devices implementing this protocol to report data / get instructions. That would make the path between device and kubernetes application much shorter than it is for other projects lilke KubeEdge whose architecture entails more elements between device and K8s application.

Many use cases show how it's used today or will be used in the future.

Describe the solution you'd like

Akri may rely on existing protocol implementations to deliver this feature rapidly

Describe alternatives you've considered

The only one known as of now is Kubeedge with the disadvantage mentioned above.

Additional context

https://dzone.com/articles/why-mqtt-has-become-the-de-facto-iot-standard

[DazWilkin]

@didier-durand I'm intrigued by this because MQTT was my initial thought on a protocol too but now, I'm unsure.

Assuming devices are able to publish to MQTT topics, what benefit do you see in writing Akri-specific protocol support for (an) MQTT (broker) over an above, running an MQTT broker in the cluster?

[didier-durand]

@DazWilkin:

Hi Daz,

This is why I suggested MQTT:

1. When I compare Akri to Kubeedge also aimed at IoT, the big difference from my perspective is the architecture: Akri tries to shorten to a very minimum the "distance" between K8s and the connected object while Kubeedge has additional layers and components, which make the architecture less homogeneous.
2. Those specific components (they include an MQTT broker) require then specific "out-of-K8s" management procedures and tools for patching, monitoring, etc.
3. So, I suggested supporting MQTT in Akri to leverage this advantage of the shortest possible to K8s. Similarly to your sayings, you could envision a solution to relay the video stream of the e2e demo (in this repo) to a remote K8s without the use of v4l2 loopback. But, the use of this solution allows working from within K8s straight away. That's the same idea for MQTT from my perspective.

This has numerous advantages coming from the normal benefits of a pure cloud-native architecture:

1. you can update the entire chain of components very fast by pushing new versions of the corresponding container images.
2. you can monitor the system end-to-end through standard K8s administration tools.
3. security will be enforced end-to-end since all components of the value chain will be under the governance of K8s
4. If you think IoT at scale, you can envision a very large number of such micro K8s clusters on the edge (all very similar to each other) to be managed by telcos (5G) or other operators: it will be very beneficial for them to be able to manage them in a federated way through tools prepared by Kubefed or equivalent.

All in all, my answer is not directly about you "why MQTT" but about why the Akri architecture is very interesting from my perspective. Then, I asked for MQTT in this ticket because it probably is by far the leader of IoT protocols: so, its support in Akri will be very beneficial to Akri & MQTT.

Please, let me know what you think of the above: I am deeply interested in different perspectives.

Didier

[jiria]

Hello @didier-durand,

Definitely agreed on how MQTT is important for IoT. Curious about your thoughts on MQTT integration to Akri. Are you suggesting to include an MQTT broker as part of Akri so leaf devices could connect to the cluster? This would still require the devices to register to the broker, not Akri finding the devices. Or are you suggesting something else? I am asking to understand how well this would fit into the Akri architecture.

[didier-durand]

@jiria :

Yes, I would definitely embark an MQTT broker as part of Akri and expose it via A k8s Service to the objects on the outside. I would do that because it would allow a quite easy (and passive) discovery of the objects based on the interception of CONNECT / CONNACK dialogs between MQTT client and broker (As per your point: that's how Akri would "find the devices" to quote you). Also, as per my answer to @DazWilkin , it would make the system more "compact", so more efficient to manage via k8s mechanisms only. Additionally, Akri would also be able to have a much tighter control of the objects allowed to connect since they could be configured (name, password) as new Akri objects.

If you keep the broker external, you will need separate mechanisms to install, patch and configure it. Also, the configuration of objects would then also be external to Akri, which wouldn't be the most efficient.

But, you can also envision a setup where the broker is pre-existant or separate for good reasons. Akri would then only be one of its subsribers. But, that's - from my perspective - a secondary option when the "embarked broker" option cannot be used.

Happy to continue the discussion if needed

Best

Didier

[jiria]

Thanks. Thinking out loud, please let me know if this is matching your thinking or if you have other thoughts. A new MQTT DiscoveryHandler would be added. This handler runs on every node (as part of or loosely coupled with Akri Agent) and would have an LB Service assigned to it that MQTT devices would connect to (this would be somewhat unique but lets see the rest). This handler would start the MQTT auth flow. We currently try not to have the DiscoveryHandler deal with secrets to reduce its complexity and improve reusability. Wonder if the auth flow Client ID could be taken here and compared against the include/exclude list in the MQTT protocol Akri Configuration. If the device is meant to be included, an MQTT broker/Akri broker is started and somehow, the device is redirected (is something like HTTP 303 possible here?) to the broker and continues its communication with the broker directly. Though this would be tricky, since the broker would be started by the Akri Controller after while and it does not make sense for DiscoveryHandler to keep the connection open until then.

This also turns Akri upside down as leaf devices would come knocking as opposed to Akri finding leaf devices. So when the Akri broker is deployed, instead of being provisioned with some information about the device that is connecting, it would be the device that would have to be provisioned with the IP of the broker.

I feel that running MQTT broker as part of DiscoveryHandler directly does not make sense, as these are running on every node and are pretty stateless at the moment.

Thoughts?

[didier-durand]

@jiria : Sorry for my delayed answer! (hectic year end..)
On your points:

1. Yes, a MQTT discovery handler should be added
2. I am not sure that it needs to run in every node: it rather needs to be attached to the MQTT broker (which can be replicated for resiliency - this implies similar replication for the handler) to listen to messages and detect new devices as they publish MQTT messages
3. This handler should filter the devices based on some YAML manifests describing the ones managed by Akri. Some others may remain out of scope for Akri.
4. I agree that authentication is probably the right way to define this filtering. This auth data should be managed as per security proposal #150 to remain efficient and coherent.
5. I agree that it somewhat turns Akri upside down due to the specifics of MQTT protocol: Akri must wait for leaf devices to show up via new messages on MQTT broker's channels. It can't try to proactively discover those devices especially since they can be remote to the host on which Akri runs. In the MQTT context, it is usual to provision the leaf device with the IP address of the MQTT broker.
6. The choice of the MQTT broker architecture is important: Akri must be able to get informed of (all ?) new messages on (all ?) new channels to define if the device comes under his management or not. So, can an existing and already containerized broker be reused or should a new one better suited to Akri's use case be developed Is a key question.
7. On the specific point of HTTP 303, I am not sure that I would rely on such an advanced HTTP redirect function for IoT devices: their IP stack may not be sophisticated enough to handle it. I would rather go in the direction of a proxy solution between the leaf devices and the effective MQTT broker if Akri doesn't provide its own implementation of MQTT combining effective broker + proxy. The proxy can be easily achieved with Service, Ingress, etc. inn K8s

I am happy to continue this thread if you've more questions or I have been unclear.

Didier

[jiria]

Sorry for delayed response from my side as well. Need a bit more time to think about it.

[shantanoo-desai]

@didier-durand and @jiria might I add to this discussion that the OASIS Standards Body also has an MQTT-SN MQTT-Sensor Netowrks Protocol v1.2

From my understanding this doesn't hamper the existing MQTT brokers but builds up an additional transparent gateways that can collect information from sensor nodes (here, Leaf Devices) and also provides a Discovery and Advertisement Protocols which can be an amazing feature for AKRI's Protocol Integration Roadmap.

I am only getting started with AKRI but MQTT is worth the effort for integration with AKRI since a very large pool of systems use this protocol.

[shantanoo-desai]

@didier-durand @jiria I have provided a draft proposal for the MQTT integration with Akri (see #341). Since the discussion has gone down for more than 6+ Months, I would like to get your opinions (if possible) on it so that we are all on the same page.

[github-actions]

Issue has been automatically marked as stale due to inactivity for 45 days. Update the issue to remove label, otherwise it will be automatically closed.

[agracey]

I think one of the big questions that I have is: how do we know when a device has gone away? Would it make sense to have a TTL where a lack of messages triggers a deletion?

[shantanoo-desai]

@agracey you could build something around the Last Will and Testament Logic available in an MQTT Broker. Docs here

[diconico07]

I looked at how MQTT is used in the wild and how we could do this MQTT discovery handler, I'll try to do a proposal for it next week (I'll likely take parts of yours @shantanoo-desai if you are okay with it).

The main points are:

• A device is a client to the MQTT broker that regularly publish to a topic
• The discovery handler is a client to the MQTT broker and subscribe to a given list of topics (with wildcards), this list is part of the discovery details
• The discovery handler can be given credentials for the MQTT broker using the discovery properties
• The ID of the device is the topic it published to (or response topic if any ?)
• There are no standard way of discovering MQTT broker, so the details of the broker are part of the discovery details
• A device "disappears" if the device go silent for an extended period of time, this timeout is part of the discovery details as it will likely vary between devices (we can't use LWT as the device might not be using this and for low power devices/sensors it is usual to not be always connected to the broker)

[josesimoes]

👍🏻 for having this! It would allow bringing in MQTT devices running on constrained micro controllers.