Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue#1804 tedge config setting for data path #1807

Merged

Conversation

albinsuresh
Copy link
Contributor

@albinsuresh albinsuresh commented Mar 10, 2023

Proposed changes

  • tedge config setting for data path
  • Use tedge data path in tedge-agent
  • Use tedge data path in c8y-config-plugin
  • Use tedge data path in c8y-firmware-plugin
  • Use tedge data path in tedge-device-management plugin

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue


#1804 #1732

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s)
  • I ran cargo fmt as mentioned in CODING_GUIDELINES
  • I used cargo clippy as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 10, 2023 12:08 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Mar 10, 2023

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass %
154 0 5 154 100

Passed Tests

Name ⏱️ Duration Suite
Define Child device 1 ID 0.057 s C8Y Child Alarms Rpi
Normal case when the child device does not exist on c8y cloud 1.9220000000000002 s C8Y Child Alarms Rpi
Normal case when the child device already exists 0.985 s C8Y Child Alarms Rpi
Reconciliation when the new alarm message arrives, restart the mapper 1.983 s C8Y Child Alarms Rpi
Reconciliation when the alarm that is cleared 5.948 s C8Y Child Alarms Rpi
Prerequisite Parent 19.937 s Child Conf Mgmt Plugin
Prerequisite Child 0.359 s Child Conf Mgmt Plugin
Child device bootstrapping 14.567 s Child Conf Mgmt Plugin
Snapshot from device 19.337 s Child Conf Mgmt Plugin
Child device config update 18.698 s Child Conf Mgmt Plugin
Configuration types should be detected on file change (without restarting service) 44.82 s Inotify Crate
Child devices support sending simple measurements 47.888 s Child Device Telemetry
Child devices support sending custom measurements 44.604 s Child Device Telemetry
Child devices support sending custom events 42.833 s Child Device Telemetry
Child devices support sending custom events overriding the type 32.96 s Child Device Telemetry
Child devices support sending custom alarms #1699 41.8 s Child Device Telemetry
Child devices support sending inventory data via c8y topic 25.432 s Child Device Telemetry
Main device support sending inventory data via c8y topic 26.075 s Child Device Telemetry
Successful firmware operation 65.147 s Firmware Operation
Install with empty firmware name 50.01 s Firmware Operation
Prerequisite Parent 20.016 s Firmware Operation Child Device
Prerequisite Child 7.844 s Firmware Operation Child Device
Child device firmware update 6.509 s Firmware Operation Child Device
Child device firmware update with cache 5.996 s Firmware Operation Child Device
Retrieve a JWT tokens 52.075 s Jwt Request
Supports restarting the device 77.534 s Restart Device
Update tedge version from previous using Cumulocity 99.042 s Tedge Self Update
Test if all c8y services are up 63.831 s Service Monitoring
Test if all c8y services are down 58.092 s Service Monitoring
Test if all c8y services are using configured service type 56.262 s Service Monitoring
Test if all c8y services using default service type when service type configured as empty 50.544 s Service Monitoring
Check health status of tedge-mapper-c8y service on broker restart 25.87 s Service Monitoring
Check health status of child device service 23.923000000000002 s Service Monitoring
Successful shell command with output 3.38 s Shell Operation
Check Successful shell command with literal double quotes output 3.171 s Shell Operation
Execute multiline shell command 3.263 s Shell Operation
Failed shell command 3.225 s Shell Operation
Software list should be populated during startup 51.096 s Software
Install software via Cumulocity 62.571 s Software
Software list should only show currently installed software and not candidates 43.755 s Software
Validate updated data path used by tedge-agent 0.898 s Data Path Config
Validate updated data path used by c8y-firmware-plugin 11.088 s Data Path Config
Stop tedge-agent service 0.441 s Log Path Config
Customize the log path 0.144 s Log Path Config
Initialize tedge-agent 0.271 s Log Path Config
Check created folders 0.156 s Log Path Config
Remove created custom folders 0.196 s Log Path Config
Install thin-edge via apt 56.547 s Install Apt
Install latest via script (from current branch) 29.86 s Install Tedge
Install specific version via script (from current branch) 16.438 s Install Tedge
Install latest tedge via script (from main branch) 23.79 s Install Tedge
Support starting and stopping services 43.083 s Service-Control
Supports a reconnect 59.23 s Test-Commands
Supports disconnect then connect 47.667 s Test-Commands
Update unknown setting 34.872 s Test-Commands
Update known setting 28.161 s Test-Commands
Stop c8y-configuration-plugin 0.242 s Health C8Y-Configuration-Plugin
Update the service file 0.216 s Health C8Y-Configuration-Plugin
Reload systemd files 0.998 s Health C8Y-Configuration-Plugin
Start c8y-configuration-plugin 0.331 s Health C8Y-Configuration-Plugin
Start watchdog service 10.427 s Health C8Y-Configuration-Plugin
Check PID of c8y-configuration-plugin 0.078 s Health C8Y-Configuration-Plugin
Kill the PID 0.118 s Health C8Y-Configuration-Plugin
Recheck PID of c8y-configuration-plugin 2.302 s Health C8Y-Configuration-Plugin
Compare PID change 0.001 s Health C8Y-Configuration-Plugin
Stop watchdog service 0.301 s Health C8Y-Configuration-Plugin
Remove entry from service file 0.28 s Health C8Y-Configuration-Plugin
Stop c8y-log-plugin 0.303 s Health C8Y-Log-Plugin
Update the service file 0.296 s Health C8Y-Log-Plugin
Reload systemd files 0.922 s Health C8Y-Log-Plugin
Start c8y-log-plugin 0.45 s Health C8Y-Log-Plugin
Start watchdog service 10.269 s Health C8Y-Log-Plugin
Check PID of c8y-log-plugin 0.216 s Health C8Y-Log-Plugin
Kill the PID 0.178 s Health C8Y-Log-Plugin
Recheck PID of c8y-log-plugin 2.21 s Health C8Y-Log-Plugin
Compare PID change 0.002 s Health C8Y-Log-Plugin
Stop watchdog service 0.146 s Health C8Y-Log-Plugin
Remove entry from service file 0.159 s Health C8Y-Log-Plugin
Stop tedge-mapper 0.346 s Health Tedge Mapper C8Y
Update the service file 0.227 s Health Tedge Mapper C8Y
Reload systemd files 0.828 s Health Tedge Mapper C8Y
Start tedge-mapper 0.376 s Health Tedge Mapper C8Y
Start watchdog service 10.413 s Health Tedge Mapper C8Y
Check PID of tedge-mapper 0.123 s Health Tedge Mapper C8Y
Kill the PID 0.161 s Health Tedge Mapper C8Y
Recheck PID of tedge-mapper 2.3449999999999998 s Health Tedge Mapper C8Y
Compare PID change 0.001 s Health Tedge Mapper C8Y
Stop watchdog service 0.412 s Health Tedge Mapper C8Y
Remove entry from service file 0.409 s Health Tedge Mapper C8Y
Stop tedge-agent 0.207 s Health Tedge-Agent
Update the service file 0.105 s Health Tedge-Agent
Reload systemd files 0.39 s Health Tedge-Agent
Start tedge-agent 0.1 s Health Tedge-Agent
Start watchdog service 10.277 s Health Tedge-Agent
Check PID of tedge-mapper 0.091 s Health Tedge-Agent
Kill the PID 0.086 s Health Tedge-Agent
Recheck PID of tedge-agent 2.214 s Health Tedge-Agent
Compare PID change 0.001 s Health Tedge-Agent
Stop watchdog service 0.122 s Health Tedge-Agent
Remove entry from service file 0.143 s Health Tedge-Agent
Stop tedge-mapper-az 0.3 s Health Tedge-Mapper-Az
Update the service file 0.333 s Health Tedge-Mapper-Az
Reload systemd files 1.3719999999999999 s Health Tedge-Mapper-Az
Start tedge-mapper-az 0.391 s Health Tedge-Mapper-Az
Start watchdog service 10.545 s Health Tedge-Mapper-Az
Check PID of tedge-mapper-az 0.156 s Health Tedge-Mapper-Az
Kill the PID 0.144 s Health Tedge-Mapper-Az
Recheck PID of tedge-agent 2.491 s Health Tedge-Mapper-Az
Compare PID change 0.001 s Health Tedge-Mapper-Az
Stop watchdog service 0.274 s Health Tedge-Mapper-Az
Remove entry from service file 0.226 s Health Tedge-Mapper-Az
Stop tedge-mapper-collectd 0.188 s Health Tedge-Mapper-Collectd
Update the service file 0.164 s Health Tedge-Mapper-Collectd
Reload systemd files 0.973 s Health Tedge-Mapper-Collectd
Start tedge-mapper-collectd 0.385 s Health Tedge-Mapper-Collectd
Start watchdog service 10.502 s Health Tedge-Mapper-Collectd
Check PID of tedge-mapper-collectd 0.166 s Health Tedge-Mapper-Collectd
Kill the PID 0.253 s Health Tedge-Mapper-Collectd
Recheck PID of tedge-mapper-collectd 0.192 s Health Tedge-Mapper-Collectd
Compare PID change 0 s Health Tedge-Mapper-Collectd
Stop watchdog service 0.266 s Health Tedge-Mapper-Collectd
Remove entry from service file 0.119 s Health Tedge-Mapper-Collectd
c8y-log-plugin health status 6.077 s MQTT health endpoints
c8y-configuration-plugin health status 5.782 s MQTT health endpoints
Wrong package name 0.154 s Improve Tedge Apt Plugin Error Messages
Wrong version 0.119 s Improve Tedge Apt Plugin Error Messages
Wrong type 0.347 s Improve Tedge Apt Plugin Error Messages
tedge_connect_test_positive 0.328 s Tedge Connect Test
tedge_connect_test_negative 0.611 s Tedge Connect Test
tedge_connect_test_sm_services 8.291 s Tedge Connect Test
tedge_disconnect_test_sm_services 2.078 s Tedge Connect Test
Install thin-edge.io 17.74 s Call Tedge
call tedge -V 0.108 s Call Tedge
call tedge -h 0.095 s Call Tedge
call tedge -h -V 0.076 s Call Tedge
call tedge help 0.096 s Call Tedge
tedge config list 0.108 s Call Tedge Config List
tedge config list --all 0.147 s Call Tedge Config List
set/unset device.type 1.176 s Call Tedge Config List
set/unset device.key.path 1.065 s Call Tedge Config List
set/unset device.cert.path 0.872 s Call Tedge Config List
set/unset c8y.root.cert.path 0.66 s Call Tedge Config List
set/unset c8y.smartrest.templates 0.94 s Call Tedge Config List
set/unset az.root.cert.path 0.699 s Call Tedge Config List
set/unset az.mapper.timestamp 0.901 s Call Tedge Config List
set/unset mqtt.bind_address 0.669 s Call Tedge Config List
set/unset mqtt.port 0.468 s Call Tedge Config List
set/unset tmp.path 0.371 s Call Tedge Config List
set/unset logs.path 0.365 s Call Tedge Config List
set/unset run.path 0.436 s Call Tedge Config List
Get Put Delete 4.65 s Http File Transfer Api
Set keys should return value on stdout 0.131 s Tedge Config Get
Unset keys should not return anything on stdout and warnings on stderr 0.502 s Tedge Config Get
Invalid keys should not return anything on stdout and warnings on stderr 0.439 s Tedge Config Get

@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from b3f8ec3 to c16e852 Compare March 13, 2023 14:16
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 13, 2023 14:23 — with GitHub Actions Inactive
@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from c16e852 to 5ecea4a Compare March 13, 2023 14:25
@albinsuresh albinsuresh marked this pull request as ready for review March 13, 2023 14:28
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 13, 2023 14:33 — with GitHub Actions Inactive
@@ -90,6 +94,7 @@ impl From<&TEdgeConfigLocation> for TEdgeConfigDefaults {
let tmp_path = Path::new(DEFAULT_TMP_PATH);
let logs_path = Path::new(DEFAULT_LOG_PATH);
let run_path = Path::new(DEFAULT_RUN_PATH);
let data_path = Path::new(DEFAULT_DATA_PATH);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not introduced by this PR, but this pattern is really heavy: a set of const used to initialize local values and to finally build a TEdgeConfigDefault struct which all fields are prefixed by a default prefix.

Why not simply Impl Default for TEdgeConfig ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I'll give that a try.

Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Happy to see that we get rid of many #[cfg(not(test))] :)

plugins/c8y_firmware_plugin/src/main.rs Outdated Show resolved Hide resolved
plugins/c8y_firmware_plugin/src/tests.rs Outdated Show resolved Hide resolved
plugins/c8y_firmware_plugin/src/tests.rs Outdated Show resolved Hide resolved
crates/core/tedge_agent/src/agent.rs Outdated Show resolved Hide resolved
crates/core/tedge_agent/src/agent.rs Outdated Show resolved Hide resolved
crates/core/tedge_agent/src/http_rest.rs Outdated Show resolved Hide resolved
plugins/c8y_configuration_plugin/src/main.rs Show resolved Hide resolved
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 14, 2023 14:59 — with GitHub Actions Inactive
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 14, 2023 15:07 — with GitHub Actions Inactive
rina23q
rina23q previously approved these changes Mar 14, 2023
Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code already looks good in my point of view.

@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 16, 2023 07:02 — with GitHub Actions Inactive
@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from 870d841 to 102aa61 Compare March 16, 2023 08:58
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file named 018_change_temp_path.md only had the documentation on updating tmp.path. I replaced it with a generic one named 018_update_config_paths.md that documents all paths that can be updated.

@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 16, 2023 09:07 — with GitHub Actions Inactive
Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments on doc.

docs/src/howto-guides/018_update_config_paths.md Outdated Show resolved Hide resolved
docs/src/howto-guides/018_update_config_paths.md Outdated Show resolved Hide resolved
@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from 66afe23 to 107d396 Compare March 16, 2023 10:54
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 16, 2023 11:02 — with GitHub Actions Inactive
@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from 107d396 to 19c7ac1 Compare March 16, 2023 13:01
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 16, 2023 13:09 — with GitHub Actions Inactive
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

Comment on lines -13 to -16
#[cfg(test)]
pub const FILE_TRANSFER_ROOT_PATH: &str = "/tmp";
#[cfg(not(test))]
pub const FILE_TRANSFER_ROOT_PATH: &str = "/var/tedge/file-transfer";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lesson to be learned with this issue & PR. Having different values for a const, while under test or not, is a sign that something is definitely wrong.

Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all good now :)

@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from 19c7ac1 to a497e8a Compare March 16, 2023 15:25
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 16, 2023 15:33 — with GitHub Actions Inactive
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 17, 2023 14:54 — with GitHub Actions Inactive
@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from 393a5ea to 622cd20 Compare March 17, 2023 15:03
@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 17, 2023 15:12 — with GitHub Actions Inactive
@albinsuresh albinsuresh force-pushed the feat/1804/configurable-data-path branch from 622cd20 to 9167d25 Compare March 20, 2023 06:51
Sleep 3s
Execute Command touch /etc/tedge/operations/c8y/${CHILD_SN}/c8y_Firmware
Sleep 3s
Cumulocity.Device Should Exist ${CHILD_SN}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make the test deterministic by removing the above sleep statements. Isn't it Cumulocity. Device Should Exist is having retries to check whether the device is already created. If not may be this needs support from the library.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately the Device Should Exist does not retry that check. The purpose of those sleeps are really not only to make sure that the device gets created, but to make sure that the device gets created with the right name. Since we're relying on inotify mechanism to get the child device created, as well as to update its supported operations, the order in which these steps are executed is critical to make sure that the device gets created with the right name (without the Mqtt device prefix) and metadata.

This definitely highlights the need for more such deterministic APIs in the Cumulocity library so that we can conditionally wait on some inventory state changes. But, it's slightly out of the scope of this ticket. So, it'd be better to address that specifically in a dedicated "test framework enhancement PR".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification. I am ok to implement this as part of another PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually Device Should Exist will work just fine there, as the external id is fine, it is just the name of the device which is not ok sometimes....But in your instance you don't really care about the name, just the external id.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also there are other checks which allow waiting for inventory fragment changes...Or are you referring to wait for a child device (which is technically a linked inventory and not a custom fragment)?

Cumulocity.Device Should Have Fragments
Cumulocity.Device Should Have Fragment Values

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glad to know that APIs like Device Should Have Fragments also exist. But from what I understood, these APIs check for the desired state only once. What we'd need on top of these existing APIs, to avoid these sleeps, is a "timed waiting" semantic, (similar to the functionality offered by this waiting package), with which we can conditionally wait for an inventory change(device created, desired fragment added etc) with a max timeout value.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general rule of thumb, all assertions are dynamic in nature, and rerun an assertion until it is true, or if the timeout interval has been reached.

Copy link
Contributor

@PradeepKiruvale PradeepKiruvale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just had a look at the integration tests, LGTM.

@albinsuresh albinsuresh temporarily deployed to Test Pull Request March 20, 2023 10:05 — with GitHub Actions Inactive
@albinsuresh albinsuresh merged commit 6a12222 into thin-edge:main Mar 20, 2023
@albinsuresh albinsuresh deleted the feat/1804/configurable-data-path branch March 20, 2023 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants