Improve integration test stability

[reubenmiller]

Proposed changes

The integration tests were ran repeatedly to try to locate sources of flakiness. Flakey tests were spotted by running the integration tests 100 times, and repeating the same 100 iterations after each set of changes to validate them. Whilst the flakiness may not be 100% removed, it is considerable improved at around 98% passing rate.

Stability improvements

• Use Kill Process keyword which will wait for a process to be killed
• Remove assertions for main device and child devices for configuration and firmware related tests. Instead this is covered by new Registration tests.
• child device configuration management: Get logs of parent device by default
• Use device time rather than test runner time for Get Unix Timestamp keyword
• Support for setting manual dns entries in the container when using the docker adapter. Settings can be added in the .env file. This is only needed if users suffer from sporadic DNS failures (e.g. when using colima on MacOS)

In addition to the stability improvements, some general improvements were made to the testing setup (discovered during the flake finder phase):

• Add new invoke flake-finder task to help find flakey integration tests
• Update library dependencies (robotframework-cumulocity and robotframework-devicelibrary)
• Improve bootstrap.sh script to make more user friendly to support running it localy during manual testing
• Add update inventory tests
• Add tests for publishing custom Cumulocity measurements
• Refactor Execute Command usage to use updated interface which allows controlling the return of stdout and stderr separately

What is not covered

The following items are not covered in this PR:

• Occasionally some tests fail if the Cumulocity IoT REST Client experiences network errors (e.g. connection reset by peer 104). These kind of issues will be covered by the library in the future to add a generic retry request logic on unexpected network issues.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)
• I ran cargo fmt as mentioned in CODING_GUIDELINES
• I used cargo clippy as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %
160	0	5	160	100

Passed Tests

Name	⏱️ Duration	Suite
Define Child device 1 ID	0.012 s	C8Y Child Alarms Rpi
Normal case when the child device does not exist on c8y cloud	2.968 s	C8Y Child Alarms Rpi
Normal case when the child device already exists	1.083 s	C8Y Child Alarms Rpi
Reconciliation when the new alarm message arrives, restart the mapper	2.458 s	C8Y Child Alarms Rpi
Reconciliation when the alarm that is cleared	6.159 s	C8Y Child Alarms Rpi
Prerequisite Parent	19.98 s	Child Conf Mgmt Plugin
Prerequisite Child	0.509 s	Child Conf Mgmt Plugin
Child device bootstrapping	16.542 s	Child Conf Mgmt Plugin
Snapshot from device	21.982 s	Child Conf Mgmt Plugin
Child device config update	17.579 s	Child Conf Mgmt Plugin
Configuration types should be detected on file change (without restarting service)	57.917 s	Inotify Crate
Child devices support sending simple measurements	2.097 s	Child Device Telemetry
Child devices support sending custom measurements	1.911 s	Child Device Telemetry
Child devices support sending custom events	1.5550000000000002 s	Child Device Telemetry
Child devices support sending custom events overriding the type	1.24 s	Child Device Telemetry
Child devices support sending custom alarms #1699	1.2570000000000001 s	Child Device Telemetry
Child devices support sending inventory data via c8y topic	1.476 s	Child Device Telemetry
Main device support sending inventory data via c8y topic	2.079 s	Child Device Telemetry
Child device supports sending custom child device measurements directly to c8y	2.366 s	Child Device Telemetry
Main device supports sending custom child device measurements directly to c8y	2.023 s	Child Device Telemetry
Successful firmware operation	71.332 s	Firmware Operation
Install with empty firmware name	68.069 s	Firmware Operation
Prerequisite Parent	23.455 s	Firmware Operation Child Device
Prerequisite Child	8.419 s	Firmware Operation Child Device
Child device firmware update	6.6370000000000005 s	Firmware Operation Child Device
Child device firmware update with cache	6.5280000000000005 s	Firmware Operation Child Device
Update Inventory data via inventory.json	3.708 s	Inventory Update
Retrieve a JWT tokens	59.888 s	Jwt Request
Main device registration	2.453 s	Device Registration
Child device registration	2.999 s	Device Registration
Supports restarting the device	86.394 s	Restart Device
Update tedge version from previous using Cumulocity	121.383 s	Tedge Self Update
Test if all c8y services are up	77.801 s	Service Monitoring
Test if all c8y services are down	75.1 s	Service Monitoring
Test if all c8y services are using configured service type	73.565 s	Service Monitoring
Test if all c8y services using default service type when service type configured as empty	44.553 s	Service Monitoring
Check health status of tedge-mapper-c8y service on broker stop start	35.189 s	Service Monitoring
Check health status of tedge-mapper-c8y service on broker restart	37.598 s	Service Monitoring
Check health status of child device service	30.424 s	Service Monitoring
Successful shell command with output	3.965 s	Shell Operation
Check Successful shell command with literal double quotes output	4.02 s	Shell Operation
Execute multiline shell command	3.828 s	Shell Operation
Failed shell command	3.535 s	Shell Operation
Software list should be populated during startup	59.379 s	Software
Install software via Cumulocity	87.257 s	Software
Software list should only show currently installed software and not candidates	57.573 s	Software
Validate updated data path used by tedge-agent	0.936 s	Data Path Config
Validate updated data path used by c8y-firmware-plugin	11.747 s	Data Path Config
Stop tedge-agent service	0.987 s	Log Path Config
Customize the log path	0.288 s	Log Path Config
Initialize tedge-agent	0.379 s	Log Path Config
Check created folders	0.33 s	Log Path Config
Remove created custom folders	0.177 s	Log Path Config
Install thin-edge via apt	51.412 s	Install Apt
Install latest via script (from current branch)	35.141 s	Install Tedge
Install specific version via script (from current branch)	24.998 s	Install Tedge
Install latest tedge via script (from main branch)	28.497 s	Install Tedge
Support starting and stopping services	58.754 s	Service-Control
Supports a reconnect	71.394 s	Test-Commands
Supports disconnect then connect	66.77 s	Test-Commands
Update unknown setting	34.642 s	Test-Commands
Update known setting	26.156 s	Test-Commands
Stop c8y-configuration-plugin	0.28 s	Health C8Y-Configuration-Plugin
Update the service file	0.467 s	Health C8Y-Configuration-Plugin
Reload systemd files	1.151 s	Health C8Y-Configuration-Plugin
Start c8y-configuration-plugin	0.328 s	Health C8Y-Configuration-Plugin
Start watchdog service	10.79 s	Health C8Y-Configuration-Plugin
Check PID of c8y-configuration-plugin	0.103 s	Health C8Y-Configuration-Plugin
Kill the PID	0.174 s	Health C8Y-Configuration-Plugin
Recheck PID of c8y-configuration-plugin	0.12 s	Health C8Y-Configuration-Plugin
Compare PID change	0.006 s	Health C8Y-Configuration-Plugin
Stop watchdog service	0.299 s	Health C8Y-Configuration-Plugin
Remove entry from service file	0.149 s	Health C8Y-Configuration-Plugin
Stop c8y-log-plugin	0.424 s	Health C8Y-Log-Plugin
Update the service file	0.468 s	Health C8Y-Log-Plugin
Reload systemd files	1.129 s	Health C8Y-Log-Plugin
Start c8y-log-plugin	0.279 s	Health C8Y-Log-Plugin
Start watchdog service	10.605 s	Health C8Y-Log-Plugin
Check PID of c8y-log-plugin	0.168 s	Health C8Y-Log-Plugin
Kill the PID	0.216 s	Health C8Y-Log-Plugin
Recheck PID of c8y-log-plugin	0.126 s	Health C8Y-Log-Plugin
Compare PID change	0.001 s	Health C8Y-Log-Plugin
Stop watchdog service	0.129 s	Health C8Y-Log-Plugin
Remove entry from service file	0.101 s	Health C8Y-Log-Plugin
Stop tedge-mapper	0.408 s	Health Tedge Mapper C8Y
Update the service file	0.346 s	Health Tedge Mapper C8Y
Reload systemd files	1.312 s	Health Tedge Mapper C8Y
Start tedge-mapper	0.281 s	Health Tedge Mapper C8Y
Start watchdog service	10.725999999999999 s	Health Tedge Mapper C8Y
Check PID of tedge-mapper	0.235 s	Health Tedge Mapper C8Y
Kill the PID	0.373 s	Health Tedge Mapper C8Y
Recheck PID of tedge-mapper	0.197 s	Health Tedge Mapper C8Y
Compare PID change	0.001 s	Health Tedge Mapper C8Y
Stop watchdog service	0.413 s	Health Tedge Mapper C8Y
Remove entry from service file	0.561 s	Health Tedge Mapper C8Y
Stop tedge-agent	0.346 s	Health Tedge-Agent
Update the service file	0.214 s	Health Tedge-Agent
Reload systemd files	0.939 s	Health Tedge-Agent
Start tedge-agent	0.369 s	Health Tedge-Agent
Start watchdog service	10.518 s	Health Tedge-Agent
Check PID of tedge-mapper	0.261 s	Health Tedge-Agent
Kill the PID	0.483 s	Health Tedge-Agent
Recheck PID of tedge-agent	0.395 s	Health Tedge-Agent
Compare PID change	0.001 s	Health Tedge-Agent
Stop watchdog service	0.291 s	Health Tedge-Agent
Remove entry from service file	0.285 s	Health Tedge-Agent
Stop tedge-mapper-az	0.28 s	Health Tedge-Mapper-Az
Update the service file	0.259 s	Health Tedge-Mapper-Az
Reload systemd files	1.087 s	Health Tedge-Mapper-Az
Start tedge-mapper-az	0.21 s	Health Tedge-Mapper-Az
Start watchdog service	10.429 s	Health Tedge-Mapper-Az
Check PID of tedge-mapper-az	0.104 s	Health Tedge-Mapper-Az
Kill the PID	0.179 s	Health Tedge-Mapper-Az
Recheck PID of tedge-agent	2.225 s	Health Tedge-Mapper-Az
Compare PID change	0.013 s	Health Tedge-Mapper-Az
Stop watchdog service	0.228 s	Health Tedge-Mapper-Az
Remove entry from service file	0.256 s	Health Tedge-Mapper-Az
Stop tedge-mapper-collectd	0.237 s	Health Tedge-Mapper-Collectd
Update the service file	0.189 s	Health Tedge-Mapper-Collectd
Reload systemd files	0.539 s	Health Tedge-Mapper-Collectd
Start tedge-mapper-collectd	0.198 s	Health Tedge-Mapper-Collectd
Start watchdog service	10.209 s	Health Tedge-Mapper-Collectd
Check PID of tedge-mapper-collectd	0.116 s	Health Tedge-Mapper-Collectd
Kill the PID	0.312 s	Health Tedge-Mapper-Collectd
Recheck PID of tedge-mapper-collectd	0.094 s	Health Tedge-Mapper-Collectd
Compare PID change	0.005 s	Health Tedge-Mapper-Collectd
Stop watchdog service	0.157 s	Health Tedge-Mapper-Collectd
Remove entry from service file	0.144 s	Health Tedge-Mapper-Collectd
c8y-log-plugin health status	6.037 s	MQTT health endpoints
c8y-configuration-plugin health status	6.083 s	MQTT health endpoints
Wrong package name	0.205 s	Improve Tedge Apt Plugin Error Messages
Wrong version	0.193 s	Improve Tedge Apt Plugin Error Messages
Wrong type	0.523 s	Improve Tedge Apt Plugin Error Messages
tedge_connect_test_positive	0.805 s	Tedge Connect Test
tedge_connect_test_negative	2.302 s	Tedge Connect Test
tedge_connect_test_sm_services	8.557 s	Tedge Connect Test
tedge_disconnect_test_sm_services	1.331 s	Tedge Connect Test
Install thin-edge.io	22.376 s	Call Tedge
call tedge -V	0.099 s	Call Tedge
call tedge -h	0.184 s	Call Tedge
call tedge -h -V	0.206 s	Call Tedge
call tedge help	0.145 s	Call Tedge
tedge config list	0.27 s	Call Tedge Config List
tedge config list --all	0.219 s	Call Tedge Config List
set/unset device.type	0.735 s	Call Tedge Config List
set/unset device.key.path	0.863 s	Call Tedge Config List
set/unset device.cert.path	0.717 s	Call Tedge Config List
set/unset c8y.root.cert.path	0.959 s	Call Tedge Config List
set/unset c8y.smartrest.templates	0.781 s	Call Tedge Config List
set/unset az.root.cert.path	1.387 s	Call Tedge Config List
set/unset az.mapper.timestamp	1.094 s	Call Tedge Config List
set/unset mqtt.bind_address	0.833 s	Call Tedge Config List
set/unset mqtt.port	0.542 s	Call Tedge Config List
set/unset tmp.path	0.372 s	Call Tedge Config List
set/unset logs.path	0.527 s	Call Tedge Config List
set/unset run.path	0.666 s	Call Tedge Config List
Get Put Delete	3.366 s	Http File Transfer Api
Set keys should return value on stdout	0.057 s	Tedge Config Get
Unset keys should not return anything on stdout and warnings on stderr	0.149 s	Tedge Config Get
Invalid keys should not return anything on stdout and warnings on stderr	0.185 s	Tedge Config Get

[didier-wenzek]

This looks good to me.

I went through all the commits (skipping quickly some of them): this is a nice set of improvements.