tests/dfu/mcuboot.test_bank_erase fails on nrf52840_pca10056

[inakypg]

commit 5b20350

***** delaying boot 1000ms (per build configuration) *****
***** Booting Zephyr OS 1.12.0-rc1 (delayed boot 1000ms) *****
Running test suite test_mcuboot_interface
===================================================================
starting test - test_bank_erase

    Assertion failed at tests/subsys/dfu/mcuboot/src/main.c:59: test_bank_erase: (pass)
fail
FAIL - test_bank_erase
===================================================================
starting test - test_request_upgrade
PASS - test_request_upgrade
===================================================================
starting test - test_write_confirm
PASS - test_write_confirm
===================================================================
===================================================================
RunID: ci-180524-0516-1661:asla
PROJECT EXECUTION FAILED

[inakypg]

First build this popped up in our radar was around 05/23 1609 (commit d73e970), but this is not necessarily the pinpoint.

[nvlsianpu]

thx, will look on that

[nvlsianpu]

@inakypg I can not reproduce this with current master (SHA 71ce9b5). Can you test it as well?
[Edit]
I can't reproduce this for v1.12.0-rc1 tag as well.

[nashif]

can't reproduce as well:

DEVICE: ***** Booting Zephyr OS v1.12.0-rc1-126-g5b20350a72 *****
DEVICE: Running test suite test_mcuboot_interface
DEVICE: ===================================================================
DEVICE: starting test - test_bank_erase
DEVICE: PASS - test_bank_erase
DEVICE: ===================================================================
DEVICE: starting test - test_request_upgrade
DEVICE: PASS - test_request_upgrade
DEVICE: ===================================================================
DEVICE: starting test - test_write_confirm
DEVICE: PASS - test_write_confirm
DEVICE: ===================================================================
DEVICE: ===================================================================
DEVICE: PROJECT EXECUTION SUCCESSFUL

[nashif]

please reopen if you still can reproduce this.

[inakypg]

Thanks for verifying, everyone

I am going to pencil this as a transient HW issue, as now I am not being able to reproduce it either, even if it failed consistently for 28 times in our environment in two different pieces of HW.

[inakypg]

I keep seeing this failing consistently in two different boards that are connected to two different test servers on a different continent each, I can't pencil this no longer to transient HW issues.

@nvlsianpu what can we do to help diagnose this remotely?

[nvlsianpu]

Is it possible that DK flash get weared? I can suppose that because they are connected to the test framework - so probably the flash is erased frequently. Did you check whether DK replacement (to brand new DK helps)?

[nvlsianpu]

nRF52840 flash erase cycle per page endurance is 10k.

[inakypg]

Thanks for your feedback -- I had missed it among the rest of the grind

This is very puzzling; I looked at all the data I have -- there are 243 confirmed failures on different pieces of hardware, all the same assertion.

It started happening on May 23rd 18 and has regularly happened until now. We've had the NRF's on the HW testing pool since before.

Once thing before I forget, that I keep forgetting, is our test infra sets CONFIG_BOOT_DELAY to 1000 (one second). Sometimes this uncovers issues.

What extra debugging can we add to get to the end of this?