[SYS] Multiple connections management (#1947)

Enable the management of different connections at runtime with a set of certs for each.
Add parameters to the WiFi Manager onboarding, client certificates, and certificate validation

.github/workflows/build.yml

@@ -62,7 +62,6 @@ jobs:
           - "nodemcuv2-rf-cc1101"
           - "nodemcuv2-somfy-cc1101"
           - "manual-wifi-test"
-          - "nodemcuv2-mqtt-fw-test"
           - "rf-wifi-gateway"
           - "nodemcuv2-rf2"
           - "nodemcuv2-rf2-cc1101"

docs/upload/advanced-configuration.md

@@ -10,16 +10,17 @@ The configuration of the broker is not covered here, you should look into the do
 
 ### Prerequisites
 The MQTT broker is configured for TLS and you have access to the CA certificate which was used to sign the MQTT broker certificate.
-You are using ESP8266 or ESP32, for other boards TLS is not supported.
+You are using ESP8266 or ESP32.
 
 ### Configure secure connection in the gateway
-To enable the secure connection and use TLS set the `#define MQTT_DEFAULT_SECURE` to true.
+To enable the secure connection and use TLS set the `#define MQTT_DEFAULT_SECURE` to true at build time, or the `mqtt secure` parameter with WiFi Manager or `mqtt_secure` with MQTT.
 Set `MQTT_SERVER` to the Common Name (CN) of the certificate of the broker.
-This can be the hostname or the IP of the broker.
+
+The server identity can be verified against a certificate or not, if you don't want to use a certicate to verify the server you can uncheck the option `validate cert` with WiFi Manager or set `mqtt_validate` to `false` with MQTT
 
 The CA certificate should be in PEM ascii format.
 If your CA certificate is not in the correct format or you don't know the format, use `openssl` to convert the certificate to the correct format.
-In `User_config.h` replace the `...` with the content of your certificate which is between the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` lines:
+At build time, in `main/certs/default_server_cert.h` replace the `...` with the content of your certificate which is between the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` lines:
 ```cpp
 const char* certificate CERT_ATTRIBUTE = R"EOF("
 -----BEGIN CERTIFICATE-----
@@ -28,7 +29,14 @@ const char* certificate CERT_ATTRIBUTE = R"EOF("
 ")EOF";
 ```
 
-You can know compile and upload to your board and the gateway should connect with TLS to your broker.
+With WiFi Manager copy your certificate from `-----BEGIN CERTIFICATE-----` to  `-----END CERTIFICATE-----` (included) and paste it into the input field:
+```
+-----BEGIN CERTIFICATE-----
+...
+-----END CERTIFICATE-----
+```
+
+This process can also be used for the other certificates, OTA, client key, client certificate if necessary.
 
 ## Add the received "value" at the end of the topic
 For the gateways that publish a "value" parameter on the json (RF, IR...), it is possible to concatenate this parameter at the end of the topic.

docs/upload/portal.md

@@ -29,8 +29,6 @@ If the board is connected by ethernet, the Wifi and password can be empty. If yo
 
 * Set your MQTT Server IP or domain name (for Home Assistant you can enter `homeassistant.local` if your server is configured with mDNS)
 * Set your MQTT Server Port (default: 1883)
-* Set the MQTT secure connection box to select whether or not the connection should be secure
-* Copy/paste the MQTT server certificate in the MQTT server cert box (only required if using a secure connection), be sure to include the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers  
 * Set your MQTT Server username (optional, 64 characters maximum)
 * Set your MQTT Server password (optional, 64 characters maximum)
 * Set your MQTT base topic if you need to change it (you must keep the / at the end) (default: home/, 64 characters maximum)
@@ -39,6 +37,10 @@ If the board is connected by ethernet, the Wifi and password can be empty. If yo
   * Local and remote OTA
   * WebUI login (login:"admin", password: "")
   * WiFi Manager Access Point password after initial config
+* Set the MQTT secure connection box to select whether or not the connection should be secure
+* Set the certificate validation to select wether or not you want to validate the server identity with a certificate
+* Copy/paste the MQTT server certificate in the MQTT server cert box (only required if using a secure connection), be sure to include the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` markers  
+* Add your certificates
 * Click on save
 
 ![WiFi manager save](../img/OpenMQTTGateway_Wifi_Manager_save.png)

docs/use/gateway.md

@@ -69,9 +69,11 @@ mosquitto_pub -t "home/OpenMQTTGateway/commands/MQTTtoSYS/config" -m
   "mqtt_pass": "password",
   "mqtt_server": "host",
   "mqtt_port": "port",
-  "mqtt_secure": "false"
+  "mqtt_validate": false,
+  "mqtt_secure": false
 }'
 ```
+
 ::: tip INFO
 By default this function is not available on the pre built binary of RFBridge, in order to have less code size and enable to have OTA update working properly. So as to enable it remove from the rf bridge env:
 ```
@@ -80,7 +82,6 @@ build_flags = '-UMQTTsetMQTT'
 :::
 
 ::: tip
-Server, port, and secure_flag are only required if changing connection to another broker.  
 If the new connection fails the gateway will fallback to the previous connection.
 :::
 
@@ -96,28 +97,103 @@ mosquitto_pub -t "home/OpenMQTTGateway/commands/MQTTtoSYS/config" -m
 This will change the subscribed and published mqtt_topic/gateway_name that the gateway uses. No parameters are mandatory, the current topic or gateway name will be used if not supplied.
 :::
 
-## Switching brokers and using self signed and client certificates
+## Switching brokers and using signed and client certificates
 
-In the `user_config.h` file it is possible to specify multiple MQTT brokers and client certificates. These are commonly self signed and are supported by defining `MQTT_SECURE_SELF_SIGNED` as true or 1.  
+In the `user_config.h` file it is possible to specify multiple MQTT brokers and client certificates.
 Additionally, support for multiple brokers and client certificates has been added. To use this, it is required that the server certificate, client certificate, and client key are provided as their own constant string value as demonstrated in the file.  
-To add more than one broker and switch between them it is necessary to provide all of the relevant certificates/keys and add their respective variable names in the `certs_array` structure, as shown in `user_config.h`, and changing the array size to the number of different connections -1.  
+To add more than one broker and switch between them it is necessary to provide all of the relevant certificates/keys and add their respective variable names in the `cnt_parameters_array` structure, as shown in `user_config.h`..  
 
-To switch between these servers with an MQTT command message, the format is as follows:
+To switch between these connections with an MQTT command message, the format is as follows:
 ```
 mosquitto_pub -t "home/OpenMQTTGateway/commands/MQTTtoSYS/config" -m
 '{
   "mqtt_user": "user",
   "mqtt_pass": "password",
   "mqtt_server": "host",
   "mqtt_port": "port",
-  "mqtt_secure": "true",
-  "mqtt_cert_index":0
+  "mqtt_secure": true,
+  "mqtt_validate": true,
+  "cnt_index":1,
+  "save_cnt": true
  }'
  ```
 ::: tip
-The `mqtt_cert_index` value corresponds to the 0 to X index of the `certs_array` in `user_config.h`.
+The `cnt_index` value corresponds to the 0 to 2 index of the `cnt_parameters_array` in `user_config.h`.
+0 being the default index, containing the onboarding parameters.
 :::
 
+To read the connection parameters:
+```
+mosquitto_pub -t "home/OpenMQTTGateway/commands/MQTTtoSYS/config" -m
+'{
+  "cnt_index":1,
+  "read_cnt": true
+ }'
+ ```
+
+ To test a connection change without saving:
+ ```
+mosquitto_pub -t "home/OpenMQTTGateway/commands/MQTTtoSYS/config" -m
+'{
+  "cnt_index":1,
+  "test_cnt": true
+ }'
+ ```
+
+## Saving/Loading connection parameters/certificates at runtime
+This chapter details the process for managing certificates/connections parameters used for secure MQTT communication with OpenMQTTGateway
+
+### Storing and Loading Certificates
+* Flash Memory Storage:
+Certificates can be saved to the flash memory using specific indices. Valid indices for storing certificates are 1 and 2, as 0 is reserved for the default certificate.
+* RAM Memory Loading:
+Certificates can be loaded from RAM, where valid indices range from 0 to 2. The device publishes a hash of the certificate to the broker to verify its identity. If the connection using the current certificate fails, the device will revert to the previous certificate.
+
+### Use Case: Changing a Group of Certificates
+When updating certificates, follow these steps to ensure that the new certificates are correctly loaded and used:
+
+1. Push Certificates via MQTT:
+Send the new certificates one by one through MQTT, using indices 1 or 2. Replace newline characters (\n) in the certificates with spaces.
+```json
+{
+  "cnt_index": 1,
+  "mqtt_server_cert": "-----BEGIN CERTIFICATE----- MIIDQTCC----END CERTIFICATE-----"
+}
+```
+
+Accepted certificates are:
+* `mqtt_server_cert`
+* `mqtt_client_cert`
+* `mqtt_client_key`
+* `ota_server_cert`
+
+2. Verify Certificates in RAM:
+After pushing the certificates, verify that they have been correctly loaded into RAM.
+```json
+{
+  "cnt_index": 1,
+  "read_cnt": true
+}
+```
+
+3. Test and Save Certificates:
+Once verification is complete, test the connection using the new certificates. If the connection is successful, send the command to save the certificates to flash.
+```json
+{
+  "cnt_index": 1,
+  "save_cnt": true
+}
+```
+
+4. Broker Connection:
+The broker will attempt to use the newly received certificates for the connection.
+
+5. Successful Connection Handling:
+If the connection is successful, the certificates are permanently stored in the flash memory at the specified index.
+
+6. Handling Connection Failures:
+If the connection fails, the device will revert to the previously used certificate index, and the new certificates will not be saved.
+
 # Firmware update from MQTT (ESP only)
 
 When the gateway used is from a standard ESP32 environment [listed and defined here](https://github.com/1technophile/OpenMQTTGateway/blob/development/environments.ini), it can be updated through a simple MQTT command:
@@ -144,7 +220,7 @@ OpenMQTTGateway checks at start and every hour if an update is available.
 
 Alternatively if you want to choose the update URL you can use the command below (ESP32 and ESP8266):
 
-Without certificate, in this case we will use the root certificate defined in User_config.h
+Without certificate, in this case the gateway will use the ota_server_cert certificate defined in default_ota_cert.h
 ```
 mosquitto_pub -t "home/OpenMQTTGateway_ESP32_BLE/commands/MQTTtoSYS/firmware_update" -m '{
   "version": "test",
@@ -153,38 +229,28 @@ mosquitto_pub -t "home/OpenMQTTGateway_ESP32_BLE/commands/MQTTtoSYS/firmware_upd
 }'
 ```
 
-With certificate:
+With certificate (replace the \n in the certificate by spaces to publish it easily):
 ```
 mosquitto_pub -t "home/OpenMQTTGateway_ESP32_BLE/commands/MQTTtoSYS/firmware_update" -m '{
   "version": "test",
   "password": "OTAPASSWORD",
   "url": "https://github.com/1technophile/OpenMQTTGateway/releases/download/v0.9.12/esp32dev-ble-firmware.bin",
-  "server_cert": "-----BEGIN CERTIFICATE-----
-MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
-MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
-d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
-QTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT
-MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
-b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG
-9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB
-CSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97
-nh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt
-43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P
-T19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4
-gdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO
-BgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR
-TLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw
-DQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr
-hMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg
-06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF
-PnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls
-YSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk
-CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
------END CERTIFICATE-----"}'
-```
-
-A bash script is available [here](ota_command_cert.zip) to simplify the use of the `server_cert` parameter.  
+  "ota_server_cert": "-----BEGIN CERTIFICATE----- MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4= -----END CERTIFICATE-----"}'
+```
+
+A bash script is available [here also](ota_command_cert.zip) to simplify the use of the `server_cert` parameter.  
+
+
+Alternatively the OTA certificate can also be saved with the cnt_index for future use:
 
+```
+mosquitto_pub -t "home/OpenMQTTGateway_ESP32_BLE/commands/MQTTtoSYS/config" -m '{
+  "cnt_index": 1,
+  "save_cnt":true,
+  "ota_server_cert": "-----BEGIN CERTIFICATE----- MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4= -----END CERTIFICATE-----"
+}'
+```
+The other connection parameters corresponding to the index need to be valid for the save function to work. This command will switch to connection parameters of index 1.
 
 To enable this functionality, `MQTT_HTTPS_FW_UPDATE` will need to be defined or the line that defines in in user_config.h will need to be uncommented.
 

environments.ini

@@ -328,9 +328,10 @@ build_flags =
   '-DZgatewayBT="BT"'
   '-DLED_SEND_RECEIVE=2'
   '-DLED_SEND_RECEIVE_ON=0'
+  '-DARDUINO_LOOP_STACK_SIZE=15000'
   '-DMQTT_SECURE_DEFAULT=true'
-  '-DMQTT_SECURE_SELF_SIGNED'
   '-DMQTT_CERT_VALIDATE_DEFAULT=true'
+  '-DMQTT_SECURE_SIGNED_CLIENT=true'
   '-DMQTT_SERVER="xxxxxxxxxxxxx-ats.iot.eu-west-2.amazonaws.com"'
   '-DMQTT_PORT="8883"'
   '-DMQTT_USER=""'
@@ -1366,18 +1367,6 @@ build_flags =
   '-DGateway_Name="OMG_TEST_MANUAL_WIFI"'
 board_build.flash_mode = dout
 
-[env:nodemcuv2-mqtt-fw-test]
-platform = ${com.esp8266_platform}
-board = nodemcuv2
-lib_deps =
-  ${com-esp.lib_deps}
-  ${libraries.wifimanager8266}
-build_flags =
-  ${com-esp.build_flags}
-  '-DMQTT_SECURE_SELF_SIGNED'
-  '-DGateway_Name="OMG_TEST_MQTT_FW"'
-board_build.flash_mode = dout
-
 [env:rf-wifi-gateway]
 platform = ${com.esp8266_platform}
 board = nodemcuv2
@@ -1748,4 +1737,5 @@ build_flags =
   '-DLED_SEND_RECEIVE_ON=0'
   '-DGateway_Name="OMG_ESP32_BLE"'
   '-DUSE_BLUFI=1'
+  '-DARDUINO_LOOP_STACK_SIZE=15000'
 custom_description = Regular BLE gateway based on esp-idf with adaptive scanning activated, automatically adapts the scan parameters depending on your devices