Absence of vestigial file system image in hex file produced by build system may cause file system corruption

[kevinjwalters]

I've been using hex files created by the build system to test bug fixes to MicroPython. I had to move the file system higher in flash due to larger code from setting a different compiler optimisation/debug level. Later, I enlarged the file system. I had a glance at the MICROBIT.hex file and I don't think it contains any data for the file system area. I believe this is unlike what's produced by the MicroPython editor in the browser as that's packaging up the python code.

The file system is fine if I just copy a few files in but if I modify them things eventually go wrong. I'm starting to suspect that the file system corruption I get after modifying files in a near full MicroPython filesystem is caused by

1. absence of code in the interpreter which thoroughly checks for the presence and integrity of the MicroPython file system in flash and
2. perhaps absence of code in the interpreter which does a full low level "format" if no valid file system is present and
3. lack of an empty file system image in MICROBIT.hex produced by the build system.

My semi-informed guess here is the find_chunk_and_erase() code can misbehave in some way when there's parts of flash that contain old bits of a file system with (nRF52 4k) pages in different places.

It looks like there's a chunk which has a marker value (single byte length) of PERSISTENT_DATA_MARKER. Is this chunk always present on flash storage? The extra space in this chunk could be used for superblock style file system metadata, at a minimum a large magic number, a version, file system length, chunk size would be useful.

(I see @jaustin has put some more general thoughts on file system stuff previously in #1 (comment) )

[kevinjwalters]

I did some testing

1. • Used the browser with https://python.microbit.org/v/3 to save the default MicroPython program to the MICROBIT drive. The file system (this is v1.18) ranges from 0x6d000 to 0x72fff.
2. • The PERSISTENT_DATA_MARKER chunk shows up in the high flash memory page.
3. • Over-wrote that with a compiled hex from a build with a 32k file system at 0x70000 to 0x77fff (note: high page has moved).
4. • The PERSISTENT_DATA_MARKER chunk is not found in either the low or high page now.

I think bad things will happen now when filesystem_sweep runs as it assumes data is correct in file system and it reads the fields where it expect PERSISTENT_DATA_MARKER values to be and then writes it to other page - this will fail if the value isn't present and bad-things-will-happen™.

[kevinjwalters]

The simplest way to improve (but not completely fix the issue described above) would be to add

config->marker = PERSISTENT_DATA_MARKER;

just before this line

micropython-microbit-v2/src/codal_port/microbitfs.c

Line 242 in 7c304ec

flash_write_bytes((uint32_t)end_page, (uint8_t*)&config, sizeof(config));

to ensure the marker value is always correct regardless of what was read from the previous block.

---

An alternative approach would be to remove the two assignments to config and replace

micropython-microbit-v2/src/codal_port/microbitfs.c

Line 218 in 7c304ec

persistent_config_t config;

with

persistent_config_t config = { .marker = PERSISTENT_DATA_MARKER };