KAD to auto-disable itself when the VUM takes over to avoid state conflicts

[vasilvas99]

Issue

KAD conflicts (overlaps) with VUM/CUA (Kanto update manager). We would like KAD to auto-disable itself when the container-update-agent (CUA) starts publishing to the containersupdate/desiredstatefeedback topic that it's identifying what actions it should take.
Ideally, once disabled the KAD service will stay disabled after reboots/updates as the UM is fully managing the device.

Solution

We add a new module mqtt_listener to KAD under a compile-time feature flag called "mqtt". This module starts a lightweight MQTT client in a separate thread that subscribes to the MQTT broker and listens on the containersupdate/desiredstatefeedback topic for messages from (UM) with status "IDENTIFIED".

Additionally the existence of a lock-file (default path: /var/lib/kanto-auto-deployer/KAD.enabled) is used as persistent marker that KAD has previously seen this message.

When the message with status "IDENTIFIED" is received:

• MQTT thread changes the extension of the lock to .disabled
• Sends a signal to the daemon (filewatcher) thread to shut down
• MQTT thread exits
• KAD exits with ERRNO=0 (success)

The systemd unit for KAD (example here) is expanded with:

[Unit]
...
ConditionPathExists=/var/lib/kanto-auto-deployer/KAD.enabled

Now systemd will always skip the KAD unit on subsequent restarts if the lock-file does not exist.

On sysVinit systems this can be checked in the runlevel-script in a similar fashion.

Testing:

1. Build QEMUx86-64 from kad-mqtt-poc of https://github.com/SoftwareDefinedVehicle/leda-distro-fork

2. Run image

3. Create a mock_desiredstatefeedback.json message (from UM tests):

{
    "activityId": "123e4567-e89b-12d3-a456-426614174000",
    "timestamp": 123456789,
    "payload": {
        "status": "INDENTIFIED",
        "message": "This is a mock desired state feedback message",
        "actions": []
    }
}

4. mosquitto_pub -t containersupdate/desiredstatefeedback -f mock_desiredstatefeedback.json

5. Check logs: journalctl -u kanto-auto-deployer. Expected:

root@qemux86-64:~# journalctl -u kanto-auto-deployer
Jun 28 11:38:30 qemux86-64 systemd[1]: Started Kanto Auto Deployer.
Jun 28 11:38:30 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:30Z INFO  kanto_auto_deployer] Running initial deployment of "/data/var/containers/manifests"
Jun 28 11:38:30 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:30Z INFO  kanto_auto_deployer] Reading manifests from [/data/var/containers/manifests]
Jun 28 11:38:30 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:30Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:30 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:30Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:30 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:30Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:30 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:30Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:31 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:31Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:31 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:31Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:32 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:32Z ERROR kanto_auto_deployer] An error occurred when connecting to socket: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [cloudconnector]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [databroker]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [feedercan]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [hvacservice-example]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [seatservice-example]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [sua]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z WARN  kanto_auto_deployer::manifest_parser] Failed to load manifest directly. Will attempt auto-conversion from init-dir format.
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Already exists [vum]
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer] Running in daemon mode. Continuously monitoring "/data/var/containers/manifests"
Jun 28 11:38:33 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:38:33Z INFO  kanto_auto_deployer::mqtt_listener] MQTT for daemon mode enabled. Will auto-disable whenever VUM takes over.
Jun 28 11:40:58 qemux86-64 kanto-auto-deployer[500]: [2023-06-28T11:40:58Z WARN  kanto_auto_deployer::fs_watcher] Getting terminated from MQTT!
Jun 28 11:40:58 qemux86-64 systemd[1]: kanto-auto-deployer.service: Deactivated successfully.
Jun 28 11:41:28 qemux86-64 systemd[1]: Kanto Auto Deployer was skipped because of a failed condition check (ConditionPathExists=/data/var/lib/kanto-auto-deployer/KAD.enabled).

Other

• Do not build the MQTT listener module in .deb releases, since this is for cases when KAD is ran as system service.
• The path of the lock can be set during compilation time by exporting the variable KAD_LOCK_PATH=/PATH/TO/LOCK/FILE to the environment and then running the cargo build
• Version has been bumped to 0.3.0
• meta-leda testing setup: Diff with eclipse-leda/meta-leda
• Leda-distro testing setup: Diff with eclipse-leda/leda-distro

[mikehaller]

Satisfies contribution guidelines, looking good for me. Approved.