Removing an operation with mv command does not remove the operation

[PradeepKiruvale]

Proposed changes

Issue:
Removing an operation file using the mv command does not remove it from the tedge-mapper's operations list.
The reason for this is when a file is moved, a file modified event gets generated and this will be considered as an add operation. So, the mapper tries to access the file and fails to read the file and add it to the list, instead of removing the operation from the list.

Proposed solution:

• So, the proposed solution is when a file is modified and a modified/Add event is generated, and if the operation file does not exist then remove the operation from the list.-
• Also replaced the operation list, with a map to store the operations. A single map is used to store the operation.-
• Populate the operations map every time there is an event, irrespective of add/remove events.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#1974

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)
• I ran cargo fmt as mentioned in CODING_GUIDELINES
• I used cargo clippy as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[didier-wenzek]

Removing an operation file using the mv command does not remove it from the list.
The reason for this is when a file is moved, then the file modified event gets generated and this will be considered as an add operation and tries to access the file and fails to read the file instead of removing the operation from the list.

I don't understand the issue. Do you mean that the Remove event is not sent to the mapper? If so, what needs to be fix is the |inotify support.

[didier-wenzek]

I just noticed that the remove_operation is buggy since the beginning.

When an operation is removed, one needs not only to remove it from the operations vector, one must also clear and rebuild the operations_by_trigger map. It's not enough to remove the deleted entry in that map, because this map points to indexes in the operations vector.

The better would be to simplify the struct Operations and use only a HashMap.

[PradeepKiruvale]

Agree, It is better to have a HashMap for operation.

[PradeepKiruvale]

Removing an operation file using the mv command does not remove it from the list.
The reason for this is when a file is moved, then the file modified event gets generated and this will be considered as an add operation and tries to access the file and fails to read the file instead of removing the operation from the list.

I don't understand the issue. Do you mean that the Remove event is not sent to the mapper? If so, what needs to be fix is the |inotify support.

Yes, when a file is removed from the /etc/tedge/operations/c8y directory using the mv command, the Remove/delete event is not sent to the mapper, instead modified event is generated, and sent to the mapper.

[PradeepKiruvale]

Removing an operation file using the mv command does not remove it from the list.
The reason for this is when a file is moved, then the file modified event gets generated and this will be considered as an add operation and tries to access the file and fails to read the file instead of removing the operation from the list.

I don't understand the issue. Do you mean that the Remove event is not sent to the mapper? If so, what needs to be fix is the |inotify support.

Do you mean to say that we need to fix it in our fs notify wrapper library or in the actual notify crate?

Yes, when a file is removed from the /etc/tedge/operations/c8y directory using the mv command, the Remove/delete event is not sent to the mapper, instead modified event is generated, and sent to the mapper.

[didier-wenzek]

Yes, when a file is removed from the /etc/tedge/operations/c8y directory using the mv command, the Remove/delete event is not sent to the mapper, instead modified event is generated, and sent to the mapper.

So that is the bug to fix.

Do you mean to say that we need to fix it in our Do you mean to say that we need to fix it in our fs notify wrapper library or in the actual notify crate? wrapper library or in the actual notify crate?

It's more than likely that the bug is in our fs-notify wrapper.

[albinsuresh]

How about a simpler approach where we don't differentiate between Add/Remove/Modified events separately but instead always reload the contents of the operations dir whenever any update happens on that directory? This prevents the operations_by_trigger map from diverging from the reality on the file system. I don't think the cost of regenerating the map by reading the whole directory content instead of selectively adding or removing from the existing map is a big deal here as adding/removing operations is such a rare operation.

[PradeepKiruvale]

How about a simpler approach where we don't differentiate between Add/Remove/Modified events separately but instead always reload the contents of the operations dir whenever any update happens on that directory? This prevents the operations_by_trigger map from diverging from the reality on the file system. I don't think the cost of regenerating the map by reading the whole directory content instead of selectively adding or removing from the existing map is a big deal here as adding/removing operations is such a rare operation.

For me, it makes sense as there will not be many operations to be loaded and the change is not that frequent. @reubenmiller @didier-wenzek what do you guys think about Albin's suggestion?

[albinsuresh]

Don't forget to remove that before the final commit.

[didier-wenzek]

Still to be removed ;-)

[PradeepKiruvale]

done

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %
214	0	5	214	100

Passed Tests

Name	⏱️ Duration	Suite
Define Child device 1 ID	0.005 s	C8Y Child Alarms Rpi
Normal case when the child device does not exist on c8y cloud	3.22 s	C8Y Child Alarms Rpi
Normal case when the child device already exists	1.213 s	C8Y Child Alarms Rpi
Reconciliation when the new alarm message arrives, restart the mapper	2.124 s	C8Y Child Alarms Rpi
Reconciliation when the alarm that is cleared	6.792 s	C8Y Child Alarms Rpi
Prerequisite Parent	21.256 s	Child Conf Mgmt Plugin
Prerequisite Child	0.837 s	Child Conf Mgmt Plugin
Child device bootstrapping	17.884 s	Child Conf Mgmt Plugin
Snapshot from device	62.174 s	Child Conf Mgmt Plugin
Child device config update	63.642 s	Child Conf Mgmt Plugin
Configuration types should be detected on file change (without restarting service)	66.801 s	Inotify Crate
Check lock file existence in default folder	1.589 s	Lock File
Check PID number in lock file	1.599 s	Lock File
Check PID number in lock file after restarting the services	2.995 s	Lock File
Check starting same service twice	3.03 s	Lock File
Switch off lock file creation	5.275 s	Lock File
Set configuration when file exists	14.846 s	Configuration Operation
Set configuration when file does not exist	6.984 s	Configuration Operation
Set configuration with broken url	8.332 s	Configuration Operation
Get configuration	7.254 s	Configuration Operation
Get non existent configuration file	5.823 s	Configuration Operation
Get non existent configuration type	6.021 s	Configuration Operation
Update configuration plugin config via cloud	6.496 s	Configuration Operation
Modify configuration plugin config via local filesystem modify inplace	3.52 s	Configuration Operation
Modify configuration plugin config via local filesystem overwrite	5.82 s	Configuration Operation
Update configuration plugin config via local filesystem copy	2.777 s	Configuration Operation
Update configuration plugin config via local filesystem move (different directory)	3.184 s	Configuration Operation
Update configuration plugin config via local filesystem move (same directory)	3.767 s	Configuration Operation
Successful firmware operation	81.65 s	Firmware Operation
Install with empty firmware name	76.715 s	Firmware Operation
Prerequisite Parent	25.43 s	Firmware Operation Child Device
Prerequisite Child	9.401 s	Firmware Operation Child Device
Child device firmware update	7.451 s	Firmware Operation Child Device
Child device firmware update with cache	7.419 s	Firmware Operation Child Device
Firmware plugin supports restart via service manager #1932	6.719 s	Firmware Operation Child Device Retry
Update Inventory data via inventory.json	2.227 s	Inventory Update
Inventory includes the agent fragment with version information	2.645 s	Inventory Update
Retrieve a JWT tokens	63.842 s	Jwt Request
Check running collectd	1.328 s	Monitor Device Collectd
Is collectd publishing MQTT messages?	3.078 s	Monitor Device Collectd
Check thin-edge monitoring	3.809 s	Monitor Device Collectd
Check grouping of measurements	9.323 s	Monitor Device Collectd
Update the custom operation dynamically	64.242 s	Dynamically Reload Operation
Main device registration	2.711 s	Device Registration
Child device registration	3.413 s	Device Registration
Supports restarting the device	88.126 s	Restart Device
Update tedge version from previous using Cumulocity	133.399 s	Tedge Self Update
Test if all c8y services are up	50.886 s	Service Monitoring
Test if all c8y services are down	124.356 s	Service Monitoring
Test if all c8y services are using configured service type	63.846 s	Service Monitoring
Test if all c8y services using default service type when service type configured as empty	252.159 s	Service Monitoring
Check health status of tedge-mapper-c8y service on broker stop start	28.738 s	Service Monitoring
Check health status of tedge-mapper-c8y service on broker restart	29.3 s	Service Monitoring
Check health status of child device service	26.537 s	Service Monitoring
Successful shell command with output	4.447 s	Shell Operation
Check Successful shell command with literal double quotes output	3.914 s	Shell Operation
Execute multiline shell command	3.634 s	Shell Operation
Failed shell command	3.973 s	Shell Operation
Software list should be populated during startup	63.152 s	Software
Install software via Cumulocity	64.286 s	Software
Software list should only show currently installed software and not candidates	65.025 s	Software
Child devices support sending simple measurements	2.674 s	Child Device Telemetry
Child devices support sending custom measurements	2.042 s	Child Device Telemetry
Child devices support sending custom events	3.848 s	Child Device Telemetry
Child devices support sending custom events overriding the type	1.509 s	Child Device Telemetry
Child devices support sending custom alarms #1699	1.579 s	Child Device Telemetry
Child devices support sending inventory data via c8y topic	3.277 s	Child Device Telemetry
Child device supports sending custom child device measurements directly to c8y	1.927 s	Child Device Telemetry
Check retained alarms	125.85 s	Raise Alarms
Thin-edge devices support sending simple measurements	3.983 s	Thin-Edge Device Telemetry
Thin-edge devices support sending simple measurements with custom type	3.542 s	Thin-Edge Device Telemetry
Thin-edge devices support sending custom measurements	1.276 s	Thin-Edge Device Telemetry
Thin-edge devices support sending custom events	3.542 s	Thin-Edge Device Telemetry
Thin-edge devices support sending custom events overriding the type	1.498 s	Thin-Edge Device Telemetry
Thin-edge devices support sending custom alarms #1699	3.573 s	Thin-Edge Device Telemetry
Thin-edge device supports sending custom Thin-edge device measurements directly to c8y	2.559 s	Thin-Edge Device Telemetry
Thin-edge device support sending inventory data via c8y topic	1.978 s	Thin-Edge Device Telemetry
thin-edge components support a custom config-dir location via flags	28.002 s	Config Dir
Validate updated data path used by tedge-agent	0.707 s	Data Path Config
Validate updated data path used by c8y-firmware-plugin	11.239 s	Data Path Config
Stop tedge-agent service	0.361 s	Log Path Config
Customize the log path	0.166 s	Log Path Config
Initialize tedge-agent	0.142 s	Log Path Config
Check created folders	0.092 s	Log Path Config
Remove created custom folders	0.117 s	Log Path Config
Install thin-edge via apt	48.788 s	Install Apt
Install latest via script (from current branch)	31.937 s	Install Tedge
Install specific version via script (from current branch)	28.44 s	Install Tedge
Install latest tedge via script (from main branch)	35.353 s	Install Tedge
Install then uninstall latest tedge via script (from main branch)	92.437 s	Install Tedge
Support starting and stopping services	48.618 s	Service-Control
Supports a reconnect	69.594 s	Test-Commands
Supports disconnect then connect	64.954 s	Test-Commands
Update unknown setting	57.163 s	Test-Commands
Update known setting	29.258 s	Test-Commands
Stop c8y-configuration-plugin	0.362 s	Health C8Y-Configuration-Plugin
Update the service file	0.377 s	Health C8Y-Configuration-Plugin
Reload systemd files	1.518 s	Health C8Y-Configuration-Plugin
Start c8y-configuration-plugin	0.273 s	Health C8Y-Configuration-Plugin
Start watchdog service	10.592 s	Health C8Y-Configuration-Plugin
Check PID of c8y-configuration-plugin	0.17 s	Health C8Y-Configuration-Plugin
Kill the PID	0.257 s	Health C8Y-Configuration-Plugin
Recheck PID of c8y-configuration-plugin	6.458 s	Health C8Y-Configuration-Plugin
Compare PID change	0.002 s	Health C8Y-Configuration-Plugin
Stop watchdog service	0.26 s	Health C8Y-Configuration-Plugin
Remove entry from service file	0.208 s	Health C8Y-Configuration-Plugin
Stop c8y-log-plugin	0.311 s	Health C8Y-Log-Plugin
Update the service file	0.283 s	Health C8Y-Log-Plugin
Reload systemd files	1.034 s	Health C8Y-Log-Plugin
Start c8y-log-plugin	0.381 s	Health C8Y-Log-Plugin
Start watchdog service	10.556000000000001 s	Health C8Y-Log-Plugin
Check PID of c8y-log-plugin	0.159 s	Health C8Y-Log-Plugin
Kill the PID	0.316 s	Health C8Y-Log-Plugin
Recheck PID of c8y-log-plugin	6.595 s	Health C8Y-Log-Plugin
Compare PID change	0.001 s	Health C8Y-Log-Plugin
Stop watchdog service	0.327 s	Health C8Y-Log-Plugin
Remove entry from service file	0.357 s	Health C8Y-Log-Plugin
Stop tedge-mapper	0.309 s	Health Tedge Mapper C8Y
Update the service file	0.209 s	Health Tedge Mapper C8Y
Reload systemd files	0.681 s	Health Tedge Mapper C8Y
Start tedge-mapper	0.184 s	Health Tedge Mapper C8Y
Start watchdog service	10.32 s	Health Tedge Mapper C8Y
Check PID of tedge-mapper	0.295 s	Health Tedge Mapper C8Y
Kill the PID	0.521 s	Health Tedge Mapper C8Y
Recheck PID of tedge-mapper	6.933 s	Health Tedge Mapper C8Y
Compare PID change	0.006 s	Health Tedge Mapper C8Y
Stop watchdog service	0.319 s	Health Tedge Mapper C8Y
Remove entry from service file	0.353 s	Health Tedge Mapper C8Y
Stop tedge-agent	0.195 s	Health Tedge-Agent
Update the service file	0.128 s	Health Tedge-Agent
Reload systemd files	0.428 s	Health Tedge-Agent
Start tedge-agent	0.231 s	Health Tedge-Agent
Start watchdog service	10.4 s	Health Tedge-Agent
Check PID of tedge-mapper	0.161 s	Health Tedge-Agent
Kill the PID	0.46 s	Health Tedge-Agent
Recheck PID of tedge-agent	6.92 s	Health Tedge-Agent
Compare PID change	0.002 s	Health Tedge-Agent
Stop watchdog service	0.516 s	Health Tedge-Agent
Remove entry from service file	0.316 s	Health Tedge-Agent
Stop tedge-mapper-az	0.185 s	Health Tedge-Mapper-Az
Update the service file	0.189 s	Health Tedge-Mapper-Az
Reload systemd files	0.711 s	Health Tedge-Mapper-Az
Start tedge-mapper-az	0.456 s	Health Tedge-Mapper-Az
Start watchdog service	10.385 s	Health Tedge-Mapper-Az
Check PID of tedge-mapper-az	0.132 s	Health Tedge-Mapper-Az
Kill the PID	0.283 s	Health Tedge-Mapper-Az
Recheck PID of tedge-agent	6.801 s	Health Tedge-Mapper-Az
Compare PID change	0.001 s	Health Tedge-Mapper-Az
Stop watchdog service	0.422 s	Health Tedge-Mapper-Az
Remove entry from service file	0.269 s	Health Tedge-Mapper-Az
Stop tedge-mapper-collectd	0.338 s	Health Tedge-Mapper-Collectd
Update the service file	0.288 s	Health Tedge-Mapper-Collectd
Reload systemd files	0.966 s	Health Tedge-Mapper-Collectd
Start tedge-mapper-collectd	0.477 s	Health Tedge-Mapper-Collectd
Start watchdog service	10.682 s	Health Tedge-Mapper-Collectd
Check PID of tedge-mapper-collectd	0.238 s	Health Tedge-Mapper-Collectd
Kill the PID	0.427 s	Health Tedge-Mapper-Collectd
Recheck PID of tedge-mapper-collectd	5.044 s	Health Tedge-Mapper-Collectd
Compare PID change	0.001 s	Health Tedge-Mapper-Collectd
Stop watchdog service	0.53 s	Health Tedge-Mapper-Collectd
Remove entry from service file	0.175 s	Health Tedge-Mapper-Collectd
tedge-collectd-mapper health status	6.095 s	Health Tedge-Mapper-Collectd
c8y-log-plugin health status	6.157 s	MQTT health endpoints
c8y-configuration-plugin health status	5.842 s	MQTT health endpoints
Publish on a local insecure broker	0.303 s	Basic Pub Sub
Publish on a local secure broker	3.705 s	Basic Pub Sub
Publish on a local secure broker with client authentication	3.5 s	Basic Pub Sub
Check remote mqtt broker #1773	5.489 s	Remote Mqtt Broker
Wrong package name	0.595 s	Improve Tedge Apt Plugin Error Messages
Wrong version	0.243 s	Improve Tedge Apt Plugin Error Messages
Wrong type	0.525 s	Improve Tedge Apt Plugin Error Messages
tedge_connect_test_positive	0.565 s	Tedge Connect Test
tedge_connect_test_negative	3.005 s	Tedge Connect Test
tedge_connect_test_sm_services	8.767 s	Tedge Connect Test
tedge_disconnect_test_sm_services	1.046 s	Tedge Connect Test
Install thin-edge.io	32.37 s	Call Tedge
call tedge -V	0.109 s	Call Tedge
call tedge -h	0.116 s	Call Tedge
call tedge -h -V	0.101 s	Call Tedge
call tedge help	0.153 s	Call Tedge
tedge config list	0.276 s	Call Tedge Config List
tedge config list --all	0.154 s	Call Tedge Config List
set/unset device.type	0.867 s	Call Tedge Config List
set/unset device.key_path	1.091 s	Call Tedge Config List
set/unset device.cert_path	1.067 s	Call Tedge Config List
set/unset c8y.root_cert_path	1.148 s	Call Tedge Config List
set/unset c8y.smartrest.templates	1.096 s	Call Tedge Config List
set/unset az.root_cert_path	1.053 s	Call Tedge Config List
set/unset aws.url	1.355 s	Call Tedge Config List
set/unset aws.root_cert_path	0.917 s	Call Tedge Config List
set/unset aws.mapper.timestamp	0.696 s	Call Tedge Config List
set/unset az.mapper.timestamp	0.578 s	Call Tedge Config List
set/unset mqtt.bind.address	0.605 s	Call Tedge Config List
set/unset mqtt.bind.port	0.535 s	Call Tedge Config List
set/unset http.bind.port	0.592 s	Call Tedge Config List
set/unset tmp.path	0.457 s	Call Tedge Config List
set/unset logs.path	0.838 s	Call Tedge Config List
set/unset run.path	0.463 s	Call Tedge Config List
set/unset firmware.child.update.timeout	0.379 s	Call Tedge Config List
set/unset c8y.url	0.372 s	Call Tedge Config List
set/unset az.url	0.554 s	Call Tedge Config List
set/unset mqtt.external.bind.port	0.418 s	Call Tedge Config List
mqtt.external.bind.address	0.372 s	Call Tedge Config List
mqtt.external.bind.interface	0.351 s	Call Tedge Config List
set/unset mqtt.external.ca_path	0.365 s	Call Tedge Config List
set/unset mqtt.external.cert_file	0.402 s	Call Tedge Config List
set/unset mqtt.external.key_file	0.326 s	Call Tedge Config List
set/unset software.plugin.default	0.337 s	Call Tedge Config List
Get Put Delete	3.108 s	Http File Transfer Api
Set keys should return value on stdout	0.083 s	Tedge Config Get
Unset keys should not return anything on stdout and warnings on stderr	0.159 s	Tedge Config Get
Invalid keys should not return anything on stdout and warnings on stderr	0.116 s	Tedge Config Get
Set configuration via environment variables	0.46 s	Tedge Config Get
Set unknown configuration via environment variables	0.085 s	Tedge Config Get

[didier-wenzek]

I'm okay to reload the whole operation set when there is a change. However, the Operations data structure must be revised. The current version is wrongly using a hashmap as a vector.

[didier-wenzek]

Approved