Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cumulocity mapper using actors #1909 #1922

Conversation

albinsuresh
Copy link
Contributor

@albinsuresh albinsuresh commented Apr 21, 2023

Proposed changes

Instead of moving the c8y mapper into extensions, I've updated the existing module itself just so that the diff is clear and easy to review. Once the changes are approved, I'll move it to extensions/c8y_mapper_ext.

  • Create actor wrapper over the CumulocityConverter
  • Refactor telemetry mapping
  • Refactor alarm sync using timer actor
  • Update telemetry tests
  • Refactor software management ops handling
  • Update software management ops mapping tests
  • Refactor custom ops handling
  • Update custom ops mapping tests
  • Fix service monitor feature
  • Move the mapper to extensions/c8y_mapper_ext.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue


Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s)
  • I ran cargo fmt as mentioned in CODING_GUIDELINES
  • I used cargo clippy as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@albinsuresh albinsuresh force-pushed the imp/1909/refactor-c8y-mapper-with-actors branch 2 times, most recently from 9419294 to c027b33 Compare April 21, 2023 05:57
@github-actions
Copy link
Contributor

github-actions bot commented Apr 21, 2023

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass %
186 0 5 186 100

Passed Tests

Name ⏱️ Duration Suite
Define Child device 1 ID 0.033 s C8Y Child Alarms Rpi
Normal case when the child device does not exist on c8y cloud 4.449 s C8Y Child Alarms Rpi
Normal case when the child device already exists 0.854 s C8Y Child Alarms Rpi
Reconciliation when the new alarm message arrives, restart the mapper 2.099 s C8Y Child Alarms Rpi
Reconciliation when the alarm that is cleared 9.082 s C8Y Child Alarms Rpi
Prerequisite Parent 79.641 s Child Conf Mgmt Plugin
Prerequisite Child 0.6 s Child Conf Mgmt Plugin
Child device bootstrapping 13.833 s Child Conf Mgmt Plugin
Snapshot from device 62.901 s Child Conf Mgmt Plugin
Child device config update 61.979 s Child Conf Mgmt Plugin
Configuration types should be detected on file change (without restarting service) 126.174 s Inotify Crate
Check lock file existence in default folder 1.336 s Lock File
Check PID number in lock file 1.468 s Lock File
Check PID number in lock file after restarting the services 2.62 s Lock File
Check starting same service twice 1.6720000000000002 s Lock File
Switch off lock file creation 4.57 s Lock File
Set configuration when file exists 55.696 s Configuration Operation
Set configuration when file does not exist 7.063 s Configuration Operation
Set configuration with broken url 8.174 s Configuration Operation
Get configuration 7.24 s Configuration Operation
Get non existent configuration file 5.66 s Configuration Operation
Get non existent configuration type 5.875 s Configuration Operation
Update configuration plugin config via cloud 5.744 s Configuration Operation
Modify configuration plugin config via local filesystem modify inplace 5.224 s Configuration Operation
Modify configuration plugin config via local filesystem overwrite 5.438 s Configuration Operation
Update configuration plugin config via local filesystem copy 3.643 s Configuration Operation
Update configuration plugin config via local filesystem move (different directory) 5.407 s Configuration Operation
Update configuration plugin config via local filesystem move (same directory) 4.115 s Configuration Operation
Successful firmware operation 87.111 s Firmware Operation
Install with empty firmware name 55.351 s Firmware Operation
Prerequisite Parent 24.149 s Firmware Operation Child Device
Prerequisite Child 8.236 s Firmware Operation Child Device
Child device firmware update 6.5440000000000005 s Firmware Operation Child Device
Child device firmware update with cache 6.097 s Firmware Operation Child Device
Update Inventory data via inventory.json 1.32 s Inventory Update
Retrieve a JWT tokens 127.166 s Jwt Request
Main device registration 2.311 s Device Registration
Child device registration 2.823 s Device Registration
Supports restarting the device 134.934 s Restart Device
Update tedge version from previous using Cumulocity 107.256 s Tedge Self Update
Test if all c8y services are up 121.375 s Service Monitoring
Test if all c8y services are down 139.512 s Service Monitoring
Test if all c8y services are using configured service type 126.355 s Service Monitoring
Test if all c8y services using default service type when service type configured as empty 101.135 s Service Monitoring
Check health status of tedge-mapper-c8y service on broker stop start 28.874 s Service Monitoring
Check health status of tedge-mapper-c8y service on broker restart 30.523 s Service Monitoring
Check health status of child device service 26.82 s Service Monitoring
Successful shell command with output 4.165 s Shell Operation
Check Successful shell command with literal double quotes output 4.113 s Shell Operation
Execute multiline shell command 3.8449999999999998 s Shell Operation
Failed shell command 3.692 s Shell Operation
Software list should be populated during startup 55.565 s Software
Install software via Cumulocity 83.334 s Software
Software list should only show currently installed software and not candidates 48.323 s Software
Child devices support sending simple measurements 4.298 s Child Device Telemetry
Child devices support sending custom measurements 1.883 s Child Device Telemetry
Child devices support sending custom events 1.722 s Child Device Telemetry
Child devices support sending custom events overriding the type 1.635 s Child Device Telemetry
Child devices support sending custom alarms #1699 1.9449999999999998 s Child Device Telemetry
Child devices support sending inventory data via c8y topic 1.2429999999999999 s Child Device Telemetry
Main device support sending inventory data via c8y topic 2.123 s Child Device Telemetry
Child device supports sending custom child device measurements directly to c8y 2.218 s Child Device Telemetry
Main device supports sending custom child device measurements directly to c8y 2.226 s Child Device Telemetry
Check retained alarms 59.814 s Raise Alarms
Validate updated data path used by tedge-agent 1.21 s Data Path Config
Validate updated data path used by c8y-firmware-plugin 13.102 s Data Path Config
Stop tedge-agent service 0.472 s Log Path Config
Customize the log path 0.173 s Log Path Config
Initialize tedge-agent 0.229 s Log Path Config
Check created folders 0.131 s Log Path Config
Remove created custom folders 0.133 s Log Path Config
Install thin-edge via apt 68.852 s Install Apt
Install latest via script (from current branch) 41.045 s Install Tedge
Install specific version via script (from current branch) 23.008 s Install Tedge
Install latest tedge via script (from main branch) 27.365 s Install Tedge
Install then uninstall latest tedge via script (from main branch) 73.583 s Install Tedge
Support starting and stopping services 56.255 s Service-Control
Supports a reconnect 75.571 s Test-Commands
Supports disconnect then connect 42.6 s Test-Commands
Update unknown setting 55.896 s Test-Commands
Update known setting 35.475 s Test-Commands
Stop c8y-configuration-plugin 60.611 s Health C8Y-Configuration-Plugin
Update the service file 0.333 s Health C8Y-Configuration-Plugin
Reload systemd files 1.311 s Health C8Y-Configuration-Plugin
Start c8y-configuration-plugin 0.373 s Health C8Y-Configuration-Plugin
Start watchdog service 10.725 s Health C8Y-Configuration-Plugin
Check PID of c8y-configuration-plugin 0.156 s Health C8Y-Configuration-Plugin
Kill the PID 0.48 s Health C8Y-Configuration-Plugin
Recheck PID of c8y-configuration-plugin 6.677 s Health C8Y-Configuration-Plugin
Compare PID change 0.001 s Health C8Y-Configuration-Plugin
Stop watchdog service 0.376 s Health C8Y-Configuration-Plugin
Remove entry from service file 0.26 s Health C8Y-Configuration-Plugin
Stop c8y-log-plugin 60.272 s Health C8Y-Log-Plugin
Update the service file 0.364 s Health C8Y-Log-Plugin
Reload systemd files 1.217 s Health C8Y-Log-Plugin
Start c8y-log-plugin 0.334 s Health C8Y-Log-Plugin
Start watchdog service 10.49 s Health C8Y-Log-Plugin
Check PID of c8y-log-plugin 0.217 s Health C8Y-Log-Plugin
Kill the PID 0.332 s Health C8Y-Log-Plugin
Recheck PID of c8y-log-plugin 6.867 s Health C8Y-Log-Plugin
Compare PID change 0.002 s Health C8Y-Log-Plugin
Stop watchdog service 0.232 s Health C8Y-Log-Plugin
Remove entry from service file 0.247 s Health C8Y-Log-Plugin
Stop tedge-mapper 0.179 s Health Tedge Mapper C8Y
Update the service file 0.118 s Health Tedge Mapper C8Y
Reload systemd files 0.267 s Health Tedge Mapper C8Y
Start tedge-mapper 0.09 s Health Tedge Mapper C8Y
Start watchdog service 10.11 s Health Tedge Mapper C8Y
Check PID of tedge-mapper 0.171 s Health Tedge Mapper C8Y
Kill the PID 0.232 s Health Tedge Mapper C8Y
Recheck PID of tedge-mapper 6.737 s Health Tedge Mapper C8Y
Compare PID change 0.005 s Health Tedge Mapper C8Y
Stop watchdog service 0.273 s Health Tedge Mapper C8Y
Remove entry from service file 0.203 s Health Tedge Mapper C8Y
Stop tedge-agent 0.254 s Health Tedge-Agent
Update the service file 0.119 s Health Tedge-Agent
Reload systemd files 0.522 s Health Tedge-Agent
Start tedge-agent 0.203 s Health Tedge-Agent
Start watchdog service 10.383 s Health Tedge-Agent
Check PID of tedge-mapper 0.122 s Health Tedge-Agent
Kill the PID 0.396 s Health Tedge-Agent
Recheck PID of tedge-agent 6.541 s Health Tedge-Agent
Compare PID change 0.002 s Health Tedge-Agent
Stop watchdog service 0.224 s Health Tedge-Agent
Remove entry from service file 0.23 s Health Tedge-Agent
Stop tedge-mapper-az 0.198 s Health Tedge-Mapper-Az
Update the service file 0.243 s Health Tedge-Mapper-Az
Reload systemd files 0.819 s Health Tedge-Mapper-Az
Start tedge-mapper-az 0.331 s Health Tedge-Mapper-Az
Start watchdog service 10.354 s Health Tedge-Mapper-Az
Check PID of tedge-mapper-az 0.099 s Health Tedge-Mapper-Az
Kill the PID 0.2 s Health Tedge-Mapper-Az
Recheck PID of tedge-agent 6.458 s Health Tedge-Mapper-Az
Compare PID change 0.002 s Health Tedge-Mapper-Az
Stop watchdog service 0.208 s Health Tedge-Mapper-Az
Remove entry from service file 0.182 s Health Tedge-Mapper-Az
Stop tedge-mapper-collectd 0.882 s Health Tedge-Mapper-Collectd
Update the service file 0.636 s Health Tedge-Mapper-Collectd
Reload systemd files 1.859 s Health Tedge-Mapper-Collectd
Start tedge-mapper-collectd 0.484 s Health Tedge-Mapper-Collectd
Start watchdog service 10.689 s Health Tedge-Mapper-Collectd
Check PID of tedge-mapper-collectd 0.18 s Health Tedge-Mapper-Collectd
Kill the PID 0.41 s Health Tedge-Mapper-Collectd
Recheck PID of tedge-mapper-collectd 6.944 s Health Tedge-Mapper-Collectd
Compare PID change 0.001 s Health Tedge-Mapper-Collectd
Stop watchdog service 0.322 s Health Tedge-Mapper-Collectd
Remove entry from service file 0.21 s Health Tedge-Mapper-Collectd
tedge-collectd-mapper health status 6.22 s Health Tedge-Mapper-Collectd
c8y-log-plugin health status 6.13 s MQTT health endpoints
c8y-configuration-plugin health status 5.949 s MQTT health endpoints
Publish on a local insecure broker 0.545 s Basic Pub Sub
Publish on a local secure broker 3.773 s Basic Pub Sub
Publish on a local secure broker with client authentication 3.76 s Basic Pub Sub
Check remote mqtt broker #1773 3.037 s Remote Mqtt Broker
Wrong package name 0.238 s Improve Tedge Apt Plugin Error Messages
Wrong version 0.131 s Improve Tedge Apt Plugin Error Messages
Wrong type 0.354 s Improve Tedge Apt Plugin Error Messages
tedge_connect_test_positive 0.245 s Tedge Connect Test
tedge_connect_test_negative 2.2640000000000002 s Tedge Connect Test
tedge_connect_test_sm_services 7.141 s Tedge Connect Test
tedge_disconnect_test_sm_services 3.268 s Tedge Connect Test
Install thin-edge.io 25.383 s Call Tedge
call tedge -V 0.197 s Call Tedge
call tedge -h 0.149 s Call Tedge
call tedge -h -V 0.107 s Call Tedge
call tedge help 0.132 s Call Tedge
tedge config list 0.086 s Call Tedge Config List
tedge config list --all 0.079 s Call Tedge Config List
set/unset device.type 0.336 s Call Tedge Config List
set/unset device.key.path 0.324 s Call Tedge Config List
set/unset device.cert.path 0.364 s Call Tedge Config List
set/unset c8y.root.cert.path 0.295 s Call Tedge Config List
set/unset c8y.smartrest.templates 0.313 s Call Tedge Config List
set/unset az.root.cert.path 0.374 s Call Tedge Config List
set/unset az.mapper.timestamp 0.34 s Call Tedge Config List
set/unset mqtt.bind_address 0.348 s Call Tedge Config List
set/unset mqtt.port 0.344 s Call Tedge Config List
set/unset tmp.path 0.353 s Call Tedge Config List
set/unset logs.path 0.348 s Call Tedge Config List
set/unset run.path 0.292 s Call Tedge Config List
Get Put Delete 4.173 s Http File Transfer Api
Set keys should return value on stdout 0.448 s Tedge Config Get
Unset keys should not return anything on stdout and warnings on stderr 0.509 s Tedge Config Get
Invalid keys should not return anything on stdout and warnings on stderr 0.404 s Tedge Config Get
Set configuration via environment variables 1.187 s Tedge Config Get
Set unknown configuration via environment variables 0.281 s Tedge Config Get

@albinsuresh albinsuresh force-pushed the imp/1909/refactor-c8y-mapper-with-actors branch from 81fdb7b to 31ff3cb Compare April 26, 2023 07:52
@albinsuresh albinsuresh force-pushed the imp/1909/refactor-c8y-mapper-with-actors branch from cab723b to 77b9c4e Compare April 26, 2023 13:15
@albinsuresh albinsuresh marked this pull request as ready for review April 26, 2023 13:16
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still have to review the tests and actor.rs . What I saw is great.

crates/core/c8y_api/src/http_proxy.rs Outdated Show resolved Hide resolved
crates/core/tedge_actors/src/message_boxes.rs Outdated Show resolved Hide resolved
crates/core/tedge_api/src/messages.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/Cargo.toml Outdated Show resolved Hide resolved
Comment on lines +113 to +115

#[error(transparent)]
FromC8YRestError(#[from] C8YRestError),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error is specific to C8Y and should not be added to core::ConversionError.

And, by the way, there is already a case for it in c8y::CumulocityMapperError.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still here - but okay the issue being beyond the scope of this PR. The core::ConversionError is already a mix of unrelated errors. This will have to be fix in a follow-up task.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, missed it. Will do it, as I'll have to resolve this anyway as part of moving the c8y-mapper into extensions/c8y_mapper_ext.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up moving the Converter trait and ConversionError enum from core to c8y_mapper_ext as no other crate was using it so it didn't make sense to keep them as separate core abstractions.

@@ -10,8 +10,10 @@ homepage = { workspace = true }
repository = { workspace = true }

[dependencies]
assert-json-diff = "2.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[follow-up task]

We need to add test-helpers feature flag.

crates/core/tedge_mapper/Cargo.toml Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/mapper.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/mapper.rs Show resolved Hide resolved
Comment on lines 10 to 11
//HIPPO Rename EventId to String as there could be many other String responses as well and this macro doesn't allow another String variant
fan_in_message_type!(C8YRestResponse[EventId, Unit, Bool]: Debug);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this macro doesn't allow another String variant

And cannot. If you receive pure Strings from different sources how can you distinguish them to process them appropriately?

fan_in_message_type!(C8YRestResponse[EventId, Unit, Bool]: Debug);

These response types are not really self contained. This is okayish for now as c8y_http_proxy is systematically used with a request-response pattern. But if for some reason we start to send concurrently several requests to the proxy, then we will have to replace these types with specific ones that give the context of the response.
An event id for what? What is true / false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you receive pure Strings from different sources how can you distinguish them to process them appropriately?

I was hoping to create a different alias like type JwtToken = String and use that aliased name as another enum variant. But I guess, both EventId and JwtToken aliases are substituted to String before the macro generates the variants, leading to the conflict. So I can either rename the existing EventId alias to just String so that it can be used everywhere or hand create the C8YRestResponse enum as follows:

enum C8YRestResponse {
    EventId(String),
    JwtToke(String),
    ...

crates/core/tedge_api/src/messages.rs Outdated Show resolved Hide resolved
crates/core/tedge_api/src/messages.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/actor.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/mapper.rs Show resolved Hide resolved
@albinsuresh albinsuresh force-pushed the imp/1909/refactor-c8y-mapper-with-actors branch from 97bf553 to 1c85072 Compare April 27, 2023 16:00
@albinsuresh albinsuresh temporarily deployed to Test Pull Request April 27, 2023 16:09 — with GitHub Actions Inactive
Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So big efforts 👍 Left some questions.

crates/core/tedge_mapper/src/c8y/tests.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/tests.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/tests.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some comments - but see nothing really blocking and would be happy to approve.

Comment on lines -42 to +44
SoftwareListRequest::topic_name(),
"tedge/commands/req/software/list"
SoftwareListRequest::topic(),
Topic::new_unchecked("tedge/commands/req/software/list")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[unrelated] I really don't see the point of this kind of tests ...

crates/core/tedge_mapper/src/c8y/converter.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/converter.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/converter.rs Outdated Show resolved Hide resolved
crates/extensions/c8y_http_proxy/src/handle.rs Outdated Show resolved Hide resolved
crates/extensions/c8y_http_proxy/src/actor.rs Outdated Show resolved Hide resolved
crates/core/tedge_mapper/src/c8y/tests.rs Outdated Show resolved Hide resolved
@albinsuresh albinsuresh temporarily deployed to Test Pull Request April 28, 2023 19:08 — with GitHub Actions Inactive
@albinsuresh albinsuresh temporarily deployed to Test Pull Request May 2, 2023 07:30 — with GitHub Actions Inactive
@albinsuresh albinsuresh temporarily deployed to Test Pull Request May 2, 2023 10:25 — with GitHub Actions Inactive
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to approve once done the final move to extensions/c8y_mapper_ext.

crates/extensions/c8y_http_proxy/src/messages.rs Outdated Show resolved Hide resolved
@albinsuresh albinsuresh temporarily deployed to Test Pull Request May 3, 2023 07:04 — with GitHub Actions Inactive
@albinsuresh
Copy link
Contributor Author

albinsuresh commented May 3, 2023

There's further cleanup that can be done in the C8y mapper code, but haven't done that in this PR as it is out-of-scope for this refactoring work. But, they'd still be worth doing in follow-up PRs. Here are a few such improvements:

  1. The different error types in c8y_mapper_ext/error.rs are quite redundant which can be cleaned up(ideally by breaking the big monolith into smaller errors)
  2. The standalone functions like parse_c8y_topics and its callees in c8y_mapper_ext/converter.rs can be moved into the CumulocityConverter struct to avoid redundant function arguments.
  3. Remove duplicated size_threshold modules from aws and az mappers and reuse the one defined in tedge_utils
  4. Guard tedge_mqtt_ext/test_helpers under a feature flag

@albinsuresh albinsuresh temporarily deployed to Test Pull Request May 3, 2023 07:39 — with GitHub Actions Inactive
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. A great step forward!

@@ -2,6 +2,7 @@ pub mod file;
pub mod fs;
pub mod paths;
pub mod signals;
pub mod size_threshold;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Note] This is typically the kind of code I see no point to abstract. This is just a size comparison but adds a coupling to the mqtt_channel crate.

@albinsuresh albinsuresh merged commit 4b6dba8 into thin-edge:main May 3, 2023
15 checks passed
@albinsuresh albinsuresh deleted the imp/1909/refactor-c8y-mapper-with-actors branch May 3, 2023 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants