Add comms (mqtt) component to forge platform

[knolleary]

Part of #464

This adds the Comms component to the platform, enabling MQTT communication.

The Comms component will enable the following uses:

• Devices reporting status and receiving commands from the platform
  • Add MQTT connectivity support device-agent#22
• Inter-project communication (EE-only)
  • Project In / Project Out nodes for interproject communication #662
• OUT OF SCOPE FOR THIS PR
  Launchers reporting status and receiving commands from the platform -
  • This PR enables the broker ACLS for the launchers to send/receive commands, but goes no further.

Broker Configuration

This requires a properly configured MQTT broker to be running and available to the platform.

We are using mosquitto, with the mosquitto-go-auth plugin.

The following is an example configuration:

auth_plugin /Users/nol/code/flowforge/mosquitto/mosquitto-go-auth/go-auth.so
auth_opt_cache true
auth_opt_backends  http
auth_opt_http_host localhost
auth_opt_http_port 3000
auth_opt_http_getuser_uri /api/comms/auth/client
auth_opt_http_aclcheck_uri /api/comms/auth/acl
per_listener_settings false
listener 4880
listener 4881
protocol websockets

Note it includes the http host/port of the forge platform, as well as the full path to the go-auth plugin.

API Routes

The mosquitto auth plugin uses http requests to the platform to authenticate and authorise requests. These are made to two new api routes:

• POST /api/comms/auth/client - validate a given username/password is valid
• POST /api/comms/auth/acl - validate an access request to a given topic is valid for a particular client

Database Changes

Adds a BrokerClient table to store username/passwords for MQTT Clients.

The platform generates its own BrokerClient on initialisation (forge_platform) - used to subscribe to status events and publish commands to devices/launchers.

Devices get a BrokerClient generated whenever they generate credentials - and the details are provided along with the other device credentials in the UI.

Runtime Configuration

A broker configuration must be provided via flowforge.yml:

broker:
  url: mqtt://localhost:1880

• Question: interval vs external broker url (ie, what launchers would use, vs what devices would use)

Next steps

• pass client credentials to launcher
  • localfs: Add FORGE_BROKER_* credentials to launcher env driver-localfs#54
  • docker: Add FORGE_BROKER_* credentials to launcher env driver-docker#30
  • k8s: Add FORGE_BROKER_* credentials to launcher env driver-k8s#44
• For all of the above PRs - need to also pass through broker connection url. How that is determined (internal vs external access)
• OUT OF SCOPE* Update launcher to use broker credentials to connect and report status - this needs to be a separate story for a future iteration. It is out of scope for the MVP.

[knolleary]

Currently reviewing whether this is the right approach for the security integration with the broker.

The problem with the dynamic security approach is we don't hold the master record of who can access what. We are reliant on the broker storing that information and it always being available.

In the k8s world, that means having a persistent volume - which complicates matters.

There is also the matter of bootstrapping its config file with the admin username/password. We wouldn't be able to mount that file in like we do flowforge.yml as mosquitto needs to write to it as well as read.

An alternative approach would be to use https://github.com/iegomez/mosquitto-go-auth which supports talking directly to sqlite/postgres to validate users.

That would allow us to keep the information in our own database - and the mosquitto instance can be more ephemeral.

The added complication is building that plugin (written in Go) for the platforms we want to support and getting it installed properly.

[knolleary]

Updated the main description to reflect the move to the go-auth plugin.

[knolleary]

Rebased this branch to main as of the 0.7 release.

[hardillb]

Looks good