MQTT Broker within FlowForge

[sammachin]

Description

To improve the efficiency of how launchers/devices report status and receive commands, we want to introduce an MQTT Broker into the FlowForge architecture. It will also enable a custom set of nodes described in #662 to provide seamless cross-project communication.

This story does not cover external usage of the broker. See #738 for the story covering that

---

• Add comms (mqtt) component to forge platform #706
• Add MQTT connectivity support device-agent#22
• Project In / Project Out nodes for interproject communication #662

[sammachin]

Some simple options could be, the Aedes node within Node-RED provides a broker inside NR
However this is not workable in FlowForge today because external TCP connections are not permitted.
Solving the TCP connection issue may be an alternative here.

[hardillb]

Solving the TCP connection issue may be an alternative here.

I believe this basically means giving each project their own IP address which I don't think i feasible (Given cost of buying IPv4 address ranges these days).

[sammachin]

doesn't have to be a whole IP address, each project could just be given a single random port.

[sammachin]

or possibly using SNI if connections were all TLS

[hardillb]

The comment was in regard to the the generic TCP access problem, not specifically MQTT.

We will need to look at the options for none HTTP load balancing (this is how K8s talks about this stuff) in AWS EKS

[PLCMercenary]

Host a broker cluster and give each project a mount point. we can surely make the subscribe connection to a broker from a project right?

[PLCMercenary]

If this issue is still being discussed, I think a simple solution would be to host a broker cluster, it doesnt matter where it is, as long as we can expand to additional compute resources. we ingress everything using a dns address, mqtt.flowforge.cloud
then just assign mount points for each admin username, or per the project. it would be simple to generate a user id on creation if the instance, and then provide a key, the mount point is then set in the backend table with a simple update and i believe the process update is a change hook for the table.

[ZJvandeWeg]

@knolleary Would appreciate your thoughts here. In general I tend to understand this as: #464 (comment)

We should ship 0.7 with a Bring your own broker, but a working version for FF.cloud.

[knolleary]

We need to be very careful to not think BYOB solves this properly.

For a FF Enterprise install, the customer can of course choose to run their own MQTT broker and use the standard MQTT nodes in NR to communicate with it. That is one valid definition of BYOB. However it does make the significant assumption that they implement their own security on the broker - controlling what credentials can publish/subscribe to what topics spaces.

When it comes to FF.cloud - we need to get the security model right from the start and it must be integrated with the Team/Project security model.

To implement the magic 'communicate between projects' nodes, we need to provide credentials for those nodes to connect to the appropriate broker. If this was running entirely inside FF.cloud (where the end user doesn't have access to the underlying credentials etc inside the container) then we could do the appropriate MQTT topic handling in the nodes to keep it secure.

The problem comes when you consider Devices - which we definitely do want to consider here. This is because the code is running outside of FF.cloud - so the credentials to connect to the broker are outside of our control. Those credentials must only allow the the client using them to use very specific MQTT topics. Otherwise it would be able to subscribe to any messages on the broker and snoop on what is being sent by anyone.

The point I'm getting to is we need much more than 'just run our own mqtt broker in ff.cloud'. We need to integrate it in such a way to be able to provision new credentials on the broker for each project and device, with suitable topic access control in place to restrict what it can do. That type of integration is going to be specific for the flavour of MQTT broker we use - and is critical to get right.

On the basis that we definitely want to go in this direction, the next steps are:

• design the topic namespace used by projects to enable cross-project communication without allowing any unintended access to other topics
• identify what broker we want to run in ff.cloud. This has to be evaluated based on:
  • ability to dynamically provision users/topic restrictions
  • able to scale (or at least, have some idea as to how we would scale in the future)

[knolleary]

Figma design doc (sharing ahead of a design working session, so currently no content to see...) - https://www.figma.com/file/BODuex2gt5E3d1KG4sXgAn/MQTT-Broker?node-id=0%3A1

[knolleary]

updated 21/7

Subscription ACLs	Can subscribe to...	Publish ACLs	Can publish to...
Platform		Platform
Receive status updates from launchers	ff/v1/+/l/+/status	Send commands to launchers	ff/v1/+/l/+/command
Receive status updates from devices	ff/v1/+/d/+/status	Send commands to devices	ff/v1/+/d/+/command
		Send commands to all devices/launchers for a project	ff/v1/+/p/+/command
Launcher/Project		Launcher/Project
Receive commands from platform	ff/v1/<team>/l/<project>/command	Send status to platform	ff/v1/<team>/l/<project>/status
Receive commands from platform	ff/v1/<team>/p/<project>/command
Receive messages from other projects	ff/v1/<team>/p/<project>/in/+/#	Send messages to other projects	ff/v1/<team>/p/+/in/+/#
Receive broadcasts from other projects	ff/v1/<team>/p/+/out/+/#	Send broadcast message	ff/v1/<team>/p/<project>/out/+/#
Device/Project		Device/Project
Receive commands from platform	ff/v1/<team>/d/<device>/command	Send status to platform	ff/v1/<team>/d/<device>/status
Receive commands from platform	ff/v1/<team>/p/<project>/command
Receive messages from other projects	ff/v1/<team>/p/<project>/in/+/#	Send messages to other projects	ff/v1/<team>/p/+/in/+/#
Receive broadcasts from other projects	ff/v1/<team>/p/+/out/+/#	Send broadcast message	ff/v1/<team>/p/<project>/out/+/#

[knolleary]

21-Jun edit - ignore this comment. We're not using mosquitto-dynamic-security, so this list of users/roles/groups is outdated.

---

Out dated comment - leaving for future reference

On the working assumption we'll use mosquitto as the broker of choice initially, we need to map the above table of topics to mosquitto users/roles/groups.

Roles

• forge_platform is the role assigned to the client the forge platform connects with.

Role	ACL Type	Topic Filter	Permission
forge_platform	subscribe	ff/v1/+/p/+/status	allow
	subscribe	ff/v1/+/d/+/status	allow
	publishClientSend	ff/v1/+/p/+/command	allow
	publishClientSend	ff/v1/+/d/+/command	allow

• team-<tid> is created for each team

Role	ACL Type	Topic Filter	Permission
team-<tid>	subscribe	ff/v1/<team>/p/+/out/+/#	allow
	publishClientSend	ff/v1/<team>/p/+/in/+/#	allow

• launcher-<pid>-admin and project-<pid> created for each project

Role	ACL Type	Topic Filter	Permission
launcher-<pid>-admin	subscribe	ff/v1/<team>/p/<project>/command	allow
	publishClientSend	ff/v1/<team>/p/<project>/status	allow
project-<pid>	subscribe	ff/v1/<team>/p/<project>/in/+/#	allow
	publishClientSend	ff/v1/<team>/p/<project>/out/+/#	allow

• device-<did>-admin created for each device in the team

Role	ACL Type	Topic Filter	Permission
device-<did>-admin	subscribe	ff/v1/<team>/d/<device>/command	allow
	publishClientSend	ff/v1/<team>/d/<device>/status	allow

Groups

For each project/device the following groups are created:

17-Jun: Current plan is to skip groups and assign roles to clients directly.

Group ID	Roles assigned to group
project-<id>	team-<tid>, launcher-<pid>-admin, project-<pid>
device-<id>	team-<tid>, device-<did>-admin, project-<pid>

[sammachin]

Just adding some notes on pricing for this:

The MQTT Broker itself will make possible 3 different user facing features:

1. Device Communication - to replace the current http polling mechanism
2. Inter project communication - between either devices and the cloud or between projects running in the cloud, this will use new custom nodes.
3. Generic user level MQTT - Using the standard nodes to communicated with other MQTT devices with thier own clients.

1 is part of devices and will not be charged for separately so should be availble to all teasm
2 should only be available to teams on certain tiers of team plan
3 is currently out of scope and not available to users

[knolleary]

A consequence of 2 is the need to introduce team tiers before we do this work. We can still do the technical work (and there is a lot of it to be done - possibly more than we can contain in one release) to add an MQTT broker to the environment for 0.7, but we won't have any new features to talk about for end-users.

[sammachin]

Yes 2 also depends on the custom nodes, #662 so there may be something to consider there, or we may want to delay that work

[knolleary]

We are shifting over to using mosquitto-go-auth for the auth plugin. This invalidates the design work above.

The table of topics is still valid (#464 (comment))

To aid with efficient ACL handling, the usernames for the clients will include the team and object id in them:

• forge_platform
• project:<teamId>:<projectId>
• device:<teamId>:<deviceId>

That allows the ACL check to verify the a project is accessing the right team topic space by comparing the team in the username with the topic component - without any db lookup needed.

In the device case, we just have to verify the device is assigned to the project it is trying to publish/subscribe on behalf of.

[hardillb]

MQTT broker URLS.

Been thinking and playing. Current thinking is that we should have 2, internal and external. Internal will be used by projects, External will be used by Devices. For all all the Container drivers it makes sense to keep the comms inside the system (both speed and cost).

Internal

• LocalFS will always be localhost (we just say the broker must be co-located with the projects/forge runtime
• Docker uses the broker container name and just works
• K8s uses broker service name and just works

No need for TLS/SSL (yet) as all running on loopback or inside the Container Orchestration system

External

• LocalFS will use the external IP/hostname of the install
• Docker/K8s will use MQTT over WS and will use mqtt. and will use the same Ingress reverse proxy as the projects.

This automatically handles WSS and means we don't need to work out how to expose arbitrary TCP services

[hardillb]

I have a working version of the helm chart that adds the broker and configures it. It exposes the broker as a http/websocket endpoint.

Will look at the docker-compose next, but it should be basically very similar.

[knolleary]

I've updated the topic table in #464 (comment) so that devices can subscribe to a common command topic for a given project - whilst allowing the launcher to have its own separate command/status topic space.

[hardillb]

Can this be closed now?