Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the Hash constraint on assert_received_unordered #1836

Merged
merged 3 commits into from
Mar 23, 2023

Conversation

didier-wenzek
Copy link
Contributor

@didier-wenzek didier-wenzek commented Mar 22, 2023

Proposed changes

  • Fix the documentation - where a missing await was making things looking fine in any case.
  • Remove the Hash constraint on the samples provided to test_helpers::assert_received_unordered
  • Add test_helpers::assert_received_matching to provide more flexibility by looking for messages matching patterns.
  • Add MessageReceiverExt::with_timeout`: to stop the tests when no message is received after a timeout
  • Add MessageReceiverExt::with_timeout`: to skip a given number of messages not relevant for a test

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue


Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s)
  • I ran cargo fmt as mentioned in CODING_GUIDELINES
  • I used cargo clippy as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request March 22, 2023 16:25 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Mar 22, 2023

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass %
154 0 5 154 100

Passed Tests

Name ⏱️ Duration Suite
Define Child device 1 ID 0.006 s C8Y Child Alarms Rpi
Normal case when the child device does not exist on c8y cloud 4.095 s C8Y Child Alarms Rpi
Normal case when the child device already exists 0.57 s C8Y Child Alarms Rpi
Reconciliation when the new alarm message arrives, restart the mapper 1.2530000000000001 s C8Y Child Alarms Rpi
Reconciliation when the alarm that is cleared 6.341 s C8Y Child Alarms Rpi
Prerequisite Parent 13.7 s Child Conf Mgmt Plugin
Prerequisite Child 0.46 s Child Conf Mgmt Plugin
Child device bootstrapping 15.451 s Child Conf Mgmt Plugin
Snapshot from device 19.13 s Child Conf Mgmt Plugin
Child device config update 18.373 s Child Conf Mgmt Plugin
Configuration types should be detected on file change (without restarting service) 47.789 s Inotify Crate
Child devices support sending simple measurements 49.298 s Child Device Telemetry
Child devices support sending custom measurements 47.59 s Child Device Telemetry
Child devices support sending custom events 45.804 s Child Device Telemetry
Child devices support sending custom events overriding the type 37.094 s Child Device Telemetry
Child devices support sending custom alarms #1699 42.071 s Child Device Telemetry
Child devices support sending inventory data via c8y topic 22.153 s Child Device Telemetry
Main device support sending inventory data via c8y topic 22.417 s Child Device Telemetry
Successful firmware operation 63.645 s Firmware Operation
Install with empty firmware name 52.525 s Firmware Operation
Prerequisite Parent 18.716 s Firmware Operation Child Device
Prerequisite Child 7.67 s Firmware Operation Child Device
Child device firmware update 6.4030000000000005 s Firmware Operation Child Device
Child device firmware update with cache 5.784 s Firmware Operation Child Device
Retrieve a JWT tokens 48.835 s Jwt Request
Supports restarting the device 76.682 s Restart Device
Update tedge version from previous using Cumulocity 90.463 s Tedge Self Update
Test if all c8y services are up 63.038 s Service Monitoring
Test if all c8y services are down 54.224 s Service Monitoring
Test if all c8y services are using configured service type 58.296 s Service Monitoring
Test if all c8y services using default service type when service type configured as empty 48.528 s Service Monitoring
Check health status of tedge-mapper-c8y service on broker restart 26.276 s Service Monitoring
Check health status of child device service 22.442 s Service Monitoring
Successful shell command with output 3.58 s Shell Operation
Check Successful shell command with literal double quotes output 3.622 s Shell Operation
Execute multiline shell command 3.096 s Shell Operation
Failed shell command 3.07 s Shell Operation
Software list should be populated during startup 50.937 s Software
Install software via Cumulocity 63.589 s Software
Software list should only show currently installed software and not candidates 45.196 s Software
Validate updated data path used by tedge-agent 1.191 s Data Path Config
Validate updated data path used by c8y-firmware-plugin 10.901 s Data Path Config
Stop tedge-agent service 0.291 s Log Path Config
Customize the log path 0.204 s Log Path Config
Initialize tedge-agent 0.153 s Log Path Config
Check created folders 0.08 s Log Path Config
Remove created custom folders 0.191 s Log Path Config
Install thin-edge via apt 50.768 s Install Apt
Install latest via script (from current branch) 28.219 s Install Tedge
Install specific version via script (from current branch) 19.595 s Install Tedge
Install latest tedge via script (from main branch) 19.968 s Install Tedge
Support starting and stopping services 43.483 s Service-Control
Supports a reconnect 54.702 s Test-Commands
Supports disconnect then connect 46.989 s Test-Commands
Update unknown setting 37.473 s Test-Commands
Update known setting 20.847 s Test-Commands
Stop c8y-configuration-plugin 0.23 s Health C8Y-Configuration-Plugin
Update the service file 0.214 s Health C8Y-Configuration-Plugin
Reload systemd files 1.069 s Health C8Y-Configuration-Plugin
Start c8y-configuration-plugin 0.255 s Health C8Y-Configuration-Plugin
Start watchdog service 10.313 s Health C8Y-Configuration-Plugin
Check PID of c8y-configuration-plugin 0.072 s Health C8Y-Configuration-Plugin
Kill the PID 0.086 s Health C8Y-Configuration-Plugin
Recheck PID of c8y-configuration-plugin 2.237 s Health C8Y-Configuration-Plugin
Compare PID change 0.014 s Health C8Y-Configuration-Plugin
Stop watchdog service 0.117 s Health C8Y-Configuration-Plugin
Remove entry from service file 0.164 s Health C8Y-Configuration-Plugin
Stop c8y-log-plugin 0.129 s Health C8Y-Log-Plugin
Update the service file 0.284 s Health C8Y-Log-Plugin
Reload systemd files 1.241 s Health C8Y-Log-Plugin
Start c8y-log-plugin 0.307 s Health C8Y-Log-Plugin
Start watchdog service 10.271 s Health C8Y-Log-Plugin
Check PID of c8y-log-plugin 0.1 s Health C8Y-Log-Plugin
Kill the PID 0.149 s Health C8Y-Log-Plugin
Recheck PID of c8y-log-plugin 2.309 s Health C8Y-Log-Plugin
Compare PID change 0.005 s Health C8Y-Log-Plugin
Stop watchdog service 0.345 s Health C8Y-Log-Plugin
Remove entry from service file 0.272 s Health C8Y-Log-Plugin
Stop tedge-mapper 0.34 s Health Tedge Mapper C8Y
Update the service file 0.291 s Health Tedge Mapper C8Y
Reload systemd files 0.844 s Health Tedge Mapper C8Y
Start tedge-mapper 0.219 s Health Tedge Mapper C8Y
Start watchdog service 10.372 s Health Tedge Mapper C8Y
Check PID of tedge-mapper 0.15 s Health Tedge Mapper C8Y
Kill the PID 0.101 s Health Tedge Mapper C8Y
Recheck PID of tedge-mapper 0.194 s Health Tedge Mapper C8Y
Compare PID change 0.004 s Health Tedge Mapper C8Y
Stop watchdog service 0.207 s Health Tedge Mapper C8Y
Remove entry from service file 0.288 s Health Tedge Mapper C8Y
Stop tedge-agent 0.277 s Health Tedge-Agent
Update the service file 0.101 s Health Tedge-Agent
Reload systemd files 0.612 s Health Tedge-Agent
Start tedge-agent 0.175 s Health Tedge-Agent
Start watchdog service 10.254 s Health Tedge-Agent
Check PID of tedge-mapper 0.109 s Health Tedge-Agent
Kill the PID 0.094 s Health Tedge-Agent
Recheck PID of tedge-agent 2.24 s Health Tedge-Agent
Compare PID change 0.001 s Health Tedge-Agent
Stop watchdog service 0.174 s Health Tedge-Agent
Remove entry from service file 0.119 s Health Tedge-Agent
Stop tedge-mapper-az 0.116 s Health Tedge-Mapper-Az
Update the service file 0.154 s Health Tedge-Mapper-Az
Reload systemd files 0.787 s Health Tedge-Mapper-Az
Start tedge-mapper-az 0.147 s Health Tedge-Mapper-Az
Start watchdog service 10.397 s Health Tedge-Mapper-Az
Check PID of tedge-mapper-az 0.18 s Health Tedge-Mapper-Az
Kill the PID 0.087 s Health Tedge-Mapper-Az
Recheck PID of tedge-agent 2.343 s Health Tedge-Mapper-Az
Compare PID change 0.001 s Health Tedge-Mapper-Az
Stop watchdog service 0.234 s Health Tedge-Mapper-Az
Remove entry from service file 0.165 s Health Tedge-Mapper-Az
Stop tedge-mapper-collectd 0.199 s Health Tedge-Mapper-Collectd
Update the service file 0.238 s Health Tedge-Mapper-Collectd
Reload systemd files 0.851 s Health Tedge-Mapper-Collectd
Start tedge-mapper-collectd 0.265 s Health Tedge-Mapper-Collectd
Start watchdog service 10.432 s Health Tedge-Mapper-Collectd
Check PID of tedge-mapper-collectd 0.168 s Health Tedge-Mapper-Collectd
Kill the PID 0.228 s Health Tedge-Mapper-Collectd
Recheck PID of tedge-mapper-collectd 0.292 s Health Tedge-Mapper-Collectd
Compare PID change 0.001 s Health Tedge-Mapper-Collectd
Stop watchdog service 0.186 s Health Tedge-Mapper-Collectd
Remove entry from service file 0.202 s Health Tedge-Mapper-Collectd
c8y-log-plugin health status 5.9190000000000005 s MQTT health endpoints
c8y-configuration-plugin health status 5.7059999999999995 s MQTT health endpoints
Wrong package name 0.25 s Improve Tedge Apt Plugin Error Messages
Wrong version 0.171 s Improve Tedge Apt Plugin Error Messages
Wrong type 0.414 s Improve Tedge Apt Plugin Error Messages
tedge_connect_test_positive 0.921 s Tedge Connect Test
tedge_connect_test_negative 2.077 s Tedge Connect Test
tedge_connect_test_sm_services 9.361 s Tedge Connect Test
tedge_disconnect_test_sm_services 1.7349999999999999 s Tedge Connect Test
Install thin-edge.io 15.41 s Call Tedge
call tedge -V 0.096 s Call Tedge
call tedge -h 0.113 s Call Tedge
call tedge -h -V 0.077 s Call Tedge
call tedge help 0.112 s Call Tedge
tedge config list 0.197 s Call Tedge Config List
tedge config list --all 0.195 s Call Tedge Config List
set/unset device.type 1.392 s Call Tedge Config List
set/unset device.key.path 1.216 s Call Tedge Config List
set/unset device.cert.path 0.593 s Call Tedge Config List
set/unset c8y.root.cert.path 0.613 s Call Tedge Config List
set/unset c8y.smartrest.templates 0.538 s Call Tedge Config List
set/unset az.root.cert.path 0.734 s Call Tedge Config List
set/unset az.mapper.timestamp 0.575 s Call Tedge Config List
set/unset mqtt.bind_address 0.505 s Call Tedge Config List
set/unset mqtt.port 0.55 s Call Tedge Config List
set/unset tmp.path 0.415 s Call Tedge Config List
set/unset logs.path 0.395 s Call Tedge Config List
set/unset run.path 0.602 s Call Tedge Config List
Get Put Delete 4.424 s Http File Transfer Api
Set keys should return value on stdout 0.182 s Tedge Config Get
Unset keys should not return anything on stdout and warnings on stderr 0.53 s Tedge Config Get
Invalid keys should not return anything on stdout and warnings on stderr 0.561 s Tedge Config Get

Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

crates/core/tedge_actors/src/test_helpers.rs Outdated Show resolved Hide resolved
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request March 22, 2023 18:23 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request March 23, 2023 11:30 — with GitHub Actions Inactive
Copy link
Contributor

@albinsuresh albinsuresh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine although I didn't fully get the motivation/benefit in moving the assert methods inside the trait.

/// A test helper that extends a message box with various way to check received messages.
#[async_trait]
pub trait MessageReceiverExt<M: Message> {
/// Return a new receiver which returns None if no message is received after the given timeout
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better if the timeout resulted in an Err rather than None? So that we can differentiate between premature channel Sender closures resulting in None response from a genuine timeout? Otherwise we'll always have to investigate both these possibilities whenever there's an assertion error, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is that the returned receiver must implement ReceiveMessages MessageReceiver.

In practice, I don't see that as an issue for testing. Either the sender has been closed prematurely while the test is expecting a value or the sender failed to send a message without the given time window. In both case, the test received nothing aka None.

This would matter if not used for testing purpose though. The ReceiveMessages MessageReceiver trait can be extended with recv method with a timeout.

/// ```
async fn assert_received<Samples>(&mut self, expected: Samples)
where
Samples: IntoIterator + Send,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why must the Samples be Send? I agree that things in the MessageBox would be Send anyway. But, just wondering if this bound was necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is necessary.

Without it, the compiler complains that "future cannot be sent between threads safely". Indeed, we iterate over the samples and await for the matching messages.

#[async_trait]
impl<T, M> MessageReceiverExt<M> for T
where
T: ReceiveMessages<M> + Send + Sync + 'static,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side commentary unrelated to this PR. I really wish this ReceiveMessages trait was called MessageReceiver.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fully agree!


async fn skip(&mut self, count: usize) {
for _ in 0..count {
let _ = self.recv().await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let _ = self.recv().await;
let _ = self.recv().with_timeout().await;

To avoid calls to skip() waiting forever?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessary as done by self ... provided self is a receiver that has been wrapped with_timeout().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice that we don't have the same understanding of the with_timeout().

  • What is provided here returns a new receiver that implicitly gives up waiting for a message on all recv() calls after some delay.
  • What you suggest here is an explicit timeout on a specific recv call.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

provided self is a receiver that has been wrapped with_timeout()

Yeah, that's one way to deal with it, sure.

I notice that we don't have the same understanding of the with_timeout()

Yeah, I imagined users using skip() on a receiver instance directly, without first wrapping it with with_timeout(). Now I understand that the users are always expected to wrap their receiver first with with_timeout() before they use this skip() API, unless they are okay with skip() waiting forever. So, yes this design is flexible for both use-cases.

Comment on lines +213 to +214
F: Fn(&T::Item, &M) -> bool,
F: Send,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
F: Fn(&T::Item, &M) -> bool,
F: Send,
F: Fn(&T::Item, &M) -> bool + Send,

I'm assuming you didn't do it this way to avoid confusion on the return type of Fn that it is bool + Send

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming you didn't do it this way to avoid confusion on the return type of Fn that it is bool + Send

Exactly. The 2 lines version seems clearer to me.

let mut received = Vec::new();

while let Some(msg) = self.recv().await {
expected.retain(|pat| !matching(pat, &msg));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a request for change but more of a personal comment: The HashSet impl was far simpler than this, especially with that !matching. I always find negation based logic difficult to follow. But I agree that the Hash constraint was a bigger limitation. So, this is fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, enforcing a Hash constraint was a pain in practice.

The main concern I have with retain is performance, leading to a received-all check with O(n^2) complexity. But this is not a problem for tests with less than 100 samples.

Comment on lines +30 to +33
/// # use tedge_actors::{Builder, NoMessage, ReceiveMessages, RuntimeError, ServiceConsumer, SimpleMessageBox, SimpleMessageBoxBuilder};
/// # use std::time::Duration;
/// # #[tokio::main]
/// # async fn main() -> Result<(),RuntimeError> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually these commented out lines won't be rendered in html. If you intended this, please ignore my comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the intent.

  • The tokio header is mandatory to use async functions in the test.
  • The use clauses seems obvious to me. But maybe, I can be wrong.

Comment on lines +202 to +203
/// |pat:&u32,msg:&MyMessage| msg.count() == *pat,
/// [3,2],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad (unrelated to PR itseld), I couldn't understand this. Can you explain what the pattern and why the expected is array..?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constraints are flexible. The expected list of samples has only to impl IntoIterator (hence, it can be an array or a vector) and the matching function has to compare each sample (provided via a reference) with a message actually received (also provided via a reference).

        Samples: IntoIterator + Send,       
        F: Fn(&Samples::Item, &M) -> bool,  

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
- with_timeout: to stop the tests on timeout
- skip: to skip a given number of messages

More helpers might be added later, as skip_while.

Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
Signed-off-by: Didier Wenzek <didier.wenzek@free.fr>
@didier-wenzek didier-wenzek temporarily deployed to Test Pull Request March 23, 2023 13:18 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek merged commit a0c3138 into thin-edge:main Mar 23, 2023
@didier-wenzek didier-wenzek deleted the actors/fix-test-helpers branch March 23, 2023 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants