tests-mbed_hal-sleep.deep-sleep K64F device fails to wake up

[studavekar]

Description

This failure is seen multiple times in CI system, created an issue to track it. May be worn out flash?

Target
K64F

Toolchain:
GCC_ARM| ARM|IAR

mbed-os sha:

9aef9d3 Merge pull request #8671 from mray19027/uart_break

Steps to reproduce

Use command run : mbed test -m K64F -t GCC_ARM -n tests-mbed_hal-sleep -v

Failure

[1542618865.26][GLRM][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1542618865.26][CONN][INF] sending up to 10 __sync packets (specified with --sync=10)
[1542618865.26][CONN][INF] sending preamble 'c956227e-4dab-4bf0-8d4e-6468ea264e28'
[1542618865.27][GLRM][TXD] {{__sync;c956227e-4dab-4bf0-8d4e-6468ea264e28}}
[1542618866.39][CONN][RXD] mbedmbedmbedmbedmbedmbedmbedmbed
[1542618866.39][CONN][INF] found SYNC in stream: {{__sync;c956227e-4dab-4bf0-8d4e-6468ea264e28}} it is #0 sent, queued...
[1542618866.39][CONN][INF] found KV pair in stream: {{__version;1.3.0}}, queued...
[1542618866.39][HTST][INF] sync KV found, uuid=c956227e-4dab-4bf0-8d4e-6468ea264e28, timestamp=1542618866.386528
[1542618866.39][HTST][INF] DUT greentea-client version: 1.3.0
[1542618866.49][CONN][INF] found KV pair in stream: {{__timeout;60}}, queued...
[1542618866.49][HTST][INF] setting timeout to: 60 sec
[1542618866.49][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued...
[1542618866.49][HTST][INF] host test class: '<class 'mbed_host_tests.host_tests.default_auto.DefaultAuto'>'
[1542618866.49][CONN][INF] found KV pair in stream: {{__testcase_count;3}}, queued...
[1542618866.49][HTST][INF] host test setup() call...
[1542618866.49][HTST][INF] CALLBACKs updated
[1542618866.49][HTST][INF] host test detected: default_auto
[1542618866.58][CONN][RXD] >>> Running 3 test cases...
[1542618866.58][CONN][INF] found KV pair in stream: {{__testcase_name;sleep - source of wake-up - us ticker}}, queued...
[1542618866.68][CONN][INF] found KV pair in stream: {{__testcase_name;deep-sleep - source of wake-up - lp ticker}}, queued...
[1542618866.78][CONN][RXD] 
[1542618866.78][CONN][RXD] >>> Running case #1: 'sleep - source of wake-up - us ticker'...
[1542618866.78][CONN][INF] found KV pair in stream: {{__testcase_name;deep-sleep - high-speed clocks are turned off}}, queued...
[1542618866.88][CONN][INF] found KV pair in stream: {{__testcase_start;sleep - source of wake-up - us ticker}}, queued...
[1542618866.98][CONN][RXD] >>> 'sleep - source of wake-up - us ticker': 1 passed, 0 failed
[1542618866.98][CONN][RXD] 
[1542618866.98][CONN][INF] found KV pair in stream: {{__testcase_finish;sleep - source of wake-up - us ticker;1;0}}, queued...
[1542618867.08][CONN][RXD] >>> Running case #2: 'deep-sleep - source of wake-up - lp ticker'...
[1542618867.18][CONN][INF] found KV pair in stream: {{__testcase_start;deep-sleep - source of wake-up - lp ticker}}, queued...
[1542618927.23][HTST][INF] test suite run finished after 60.75 sec...
[1542618927.23][CONN][INF] received special event '__host_test_finished' value='True', finishing
[1542618927.24][urllib3.connectionpool]Starting new HTTP connection (1): austin-ci-linux-001:8000
[1542618927.28][urllib3.connectionpool]http://austin-ci-linux-001:8000 "PUT /resource/0240000040214e4500341018c892003e9e11000097969900/disconnect HTTP/1.1" 200 2
[1542618927.28][urllib3.connectionpool]Starting new HTTP connection (1): austin-ci-linux-001:8000
[1542618930.41][urllib3.connectionpool]http://austin-ci-linux-001:8000 "PUT /resource/0240000040214e4500341018c892003e9e11000097969900/release HTTP/1.1" 200 66
[1542618930.42][HTST][INF] CONN exited with code: 0
[1542618930.42][HTST][INF] No events in queue
[1542618930.42][HTST][INF] stopped consuming events
[1542618930.42][HTST][INF] host test result(): None
[1542618930.42][HTST][WRN] missing __exit event from DUT
[1542618930.42][HTST][WRN] missing __exit_event_queue event from host test
[1542618930.42][HTST][ERR] missing __exit_event_queue event from host test and no result from host test, timeout...
[1542618930.42][HTST][INF] calling blocking teardown()
[1542618930.42][HTST][INF] teardown() finished
[1542618930.42][HTST][INF] {{result;timeout}}

Issue request type

[ ] Question
[ ] Enhancement
[ X ] Bug

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-182

[0xc0170]

cc @ARMmbed/team-nxp

[0xc0170]

I reviewed few recent failures. Most of them indicate (not sure why CPU info test fails ?) something is wrong with the tickers - cant wake up from sleep (rtc/lpticker or sleep tests). They end up in the timeout when invoked sleep.

Reviewing latest changes to targets, there was this fix for RTC init: #8272 (it affects rtc and lp ticker, both related to our tests failing). I reviewed the code, looks good to me (follows the application note referenced there). The change was accepted some time ago (a month ago in the 5.10.2 release).

To the question : worn out flash? FlashIAP was enabled for KL82, KW41, K22 (#8310), also for 5.10.2

Note, I can't reproduce locally. What steps should I follow ? I fetched binaries from failed build or rebuild from master, passing on K64F. I tried to run few times in row, restart device after each run, so far no luck.

cc @fkjagodzinski You confirmed this fix, have you experienced any issues we are seeing now?

[deepikabhavnani]

#8272 discusses about fix working for certain versions of GCC ARM toolchain.. Failure is seen for GCC only?

[0xc0170]

I wish. ARM and also IAR affected, see 3456 or earlier test jobs. I can try to revert it and run through tests (to see what impact it has).

[fkjagodzinski]

cc @fkjagodzinski You confirmed this fix, have you experienced any issues we are seeing now?

The issue I had reported was fixed with #8272. No problems when I checked with the code from the issue for all three toolchains, #5348 (comment). I didn't run any lp_ticker or sleep tests back then, but morph tests passed on that PR.

#8272 discusses about fix working for certain versions of GCC ARM toolchain.. Failure is seen for GCC only?

Please note that comment regards only the GCC v7, withch is not supported by the Mbed OS.

[studavekar]

Mbed os hardware tests are passing in CI closing this issue.