Silent file corruption on NRF devices

[kdb424]

On large disk writes, filesystem corruption seems to be an issue on the disk. It reports as working properly to the host OS, even after a reboot of the controller. This has been tested on several devices, but is most problematic on the nice_nano, which has almost identical specs to the itsybitsyNRF. Tested using several Cpy versions including 6 rc.0. Steps to reproduce
https://github.com/KMKfw/kmk_firmware

1. copy KMK folder to the root of the drive
2. wait until disk sync or run sync on linux
3. code is corrupted when trying to run

This issue is much more likely to occur while trying to replace files on disk, though even initial copy seems to fail, and half of the time, crashes to safe mode if a disk sync is forced. rsync reports that the files are correct, when the code is run, random files will have errors that make no sense, and be corrupt with garbage data. Copying the same files over them works eventually, and all code with 0 changes will run properly.

Tested on Linux, as well as a Windows 10 VM on Virtualbox

[dhalbert]

The nice_nano does not have an external QSPI flash chip, and uses the nRF52840's internal flash for a filesystem. On the Particle Xenon, CIRCUITPY is on the external flash chip.

Do you see any difference between boards that use use the internal flash and external flash when you see this problem?

[kdb424]

I remembered incorrectly. Sorry for that mistake, the Xenon works perfectly.

[dhalbert]

OK, so this is probably internal filesystem only. Thanks! That's very helpful. We have seen this in the past, and I have a vague memory of trying to fix it, but clearly it needs more investigation.

[dhalbert]

Some previous issues with similar problems that we thought we fixed:
#2318
#2338
#1654

[kdb424]

Let me know when you think you have a fix, let me know the branch and I'll give it a shot on my devices as well and report back. I don't have nearly a deep enough understanding of circuitpython to contribute farther.

[tannewt]

@dhalbert isn't this a bug? I'd expect the bug label and either the 6.x.x milestone if we're going to debug soon or Long Term if we aren't.

[dhalbert]

It is a bug that I marked it with the wrong milestone :)

[kdb424]

Wanted to follow this up with tests on Mac and Windows. Same results. Success reported when writing the file, safely ejected the disk. File corruption detected when running code as there were errors on the code that was read back on the nice_nano itself. Rebooting the microcontroller (hard reset), the corruption could be seen in that file on the computer.

Edit: Looks like I may have spoken too soon. It may be writing correctly, and reporting to the OS that it's writing faster than it is. I left the board for 5 minutes after the copy, and eventually it seemed to settle if I didn't interrupt it. Running sync on the system claimed that it was done writing while the code still would not run. Doing nothing for 5 minutes as the code tried to run over and over eventually worked. Will report back any further findings.

[kdb424]

Following up that this still is an issue.

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
main.py output:
Traceback (most recent call last):
  File "main.py", line 8, in <module>
  File "0İ4|.df2�3��22�3��22[_init_coord_mappingkmk/kmk_keyboard.pyany
extensionsany
co", line 4, in <module>
RuntimeError: Corrupt .mpy file

Clean formatting storage with storage.erase_filesystem seems to help for a while. Eventually I have to run that or all files are corrupt and the device will disconnect me from a comm (REPL) any time I copy a file.

[kdb424]

I've managed to come up with a way to consistently get corruption. I've frozen in as many modules as I could to fill up the primary storage. My custom circuitpython image is at 1.1MB. Adding only a few small files onto the CIRCUITPY drive and I consistently see corruption very often. Hopefully this will help you reproduce this more quickly to see what is causing the issue. I can send over the U2F for the nice_nano as well that I used to trigger this as well if it is at all helpful.