Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Host test timing_drift_auto can return passing result even if crash occurs #5256

Closed
c1728p9 opened this issue Oct 4, 2017 · 1 comment
Closed

Comments

@c1728p9
Copy link
Contributor

c1728p9 commented Oct 4, 2017

The host test timing_drift_auto returns success based on the individual test cases run, but not on the test as a whole. Because of this, if a crash occurs during teardown after all tests have run and passed then the host test will return pass, even though a crash occurred.

An example of this can be seen on the nightly results for PR #5242:

[1507071477.06][HTST][INF] DUT end time  : 16964000.0
[1507071477.06][HTST][INF] min_pass : 16.2592500091 , max_pass : 17.6001000881 for 5.0%%
[1507071477.06][HTST][INF] min_inconclusive : 15.9239000797 , max_inconclusive : 17.97075001
[1507071477.06][HTST][INF] Time reported by device: 16.898
[1507071477.06][HTST][INF] Test passed !!!
[1507071477.07][SERI][TXD] {{pass;0}}
[1507071477.19][CONN][RXD] Thread 0x0 error -6: Not allowed in ISR context
[1507071690.48][HTST][INF] test suite run finished after 240.53 sec...
[1507071690.48][CONN][INF] received special even '__host_test_finished' value='True', finishing
[1507071690.51][HTST][INF] CONN exited with code: 0
[1507071690.51][HTST][INF] No events in queue
[1507071690.51][HTST][INF] stopped consuming events
[1507071690.51][HTST][INF] host test result(): True
[1507071690.51][HTST][WRN] missing __exit event from DUT
[1507071690.51][HTST][WRN] missing __exit_event_queue event from host test
[1507071690.51][HTST][INF] calling blocking teardown()
[1507071690.51][HTST][INF] teardown() finished
[1507071690.51][HTST][INF] {{result;success}}
@c1728p9
Copy link
Contributor Author

c1728p9 commented Oct 4, 2017

cc @studavekar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants