RP2040 (Raspberry Pi Pico W) - CIRCUITPY appears corrupted on linux

[epeay]

CircuitPython version

I flashed 8.2.7, and then bounced back to 8.2.2. Same issue on both versions.

Code/REPL

N/A

Behavior

I have a Raspberry Pi Pico W.

When I flash micropython, I have no trouble. This only happens with circuitpython.

To Recreate:

1. Hold BOOTSEL, unplug and re-plug the pico W. Release BOOTSEL.
2. Copy the 8.2.7 bootloader onto the RPI... mount.
3. Wait for the flash to finish. The drive unmounts. The new CIRCUITPY mount is created.
4. Attempt to view the files in the mount.

Expected:

A normal-ish filesystem.

Actual:

> /media/epeay/CIRCUITPY$ ll
ls: cannot access '.fseventsd': Input/output error
ls: cannot access 'lib': Input/output error
ls: cannot access 't pin  #. Im': Input/output error
ls: cannot access 'or GPIO.con': Input/output error
ls: cannot access 't red'$'\n''im.por': Input/output error
ls: cannot access 'in=0'$'\n''boa.RD_': Input/output error
ls: cannot access 'LS  # Nu.mbe': Input/output error
ls: cannot access ' leds =.mat': Input/output error
ls: cannot access 'eoPixel.cla': Input/output error
ls: cannot access 'onColor.= (': Input/output error
ls: cannot access 't pin  #. Im': Input/output error
ls: cannot access 'or GPIO.con': Input/output error
ls: cannot access 't red'$'\n''im.por': Input/output error

This is junk data, being read as a filesystem.

Description

I have the same symptoms as #2944 , and the workaround mentioned in that bug, also worked for me.

1. Unplug the pico W.
2. Plug into a mac. Observe the correctly mounted filesystem.
3. Unplug from the mac.
4. Plug back into linux. Observe the correctly mounted filesystem.

> /media/epeay/CIRCUITPY$ ll
total 26
drwxr-xr-x  3 epeay epeay 16384 Dec 31  1969 ./
drwxr-x---+ 6 root  root   4096 Oct 28 23:28 ../
-rw-r--r--  1 epeay epeay  4096 Oct 28 16:25 ._boot_out.txt
-rw-r--r--  1 epeay epeay   152 Dec 31  2019 boot_out.txt
-rw-r--r--  1 epeay epeay    22 Dec 31  2019 code.py
drwxr-xr-x  2 epeay epeay   512 Oct 28 16:25 .fseventsd/
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 .metadata_never_index
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 settings.toml
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 .Trashes

Don't know why that workaround works, but it does.

Additional information

No response

[dhalbert]

What version of macOS are you using?

Have you tried to reformat the filesystem (after saving your work, if necessary):

>>> import storage
>>> storage.erase_filesystem()

Then write some files on one or the other. Then, does it work on both Mac and Linux?

[RetiredWizard]

I'm running Ubuntu and wasn't able to reproduce this either by reflashing CP 8.2.7 or by flashing Micropython, filling the file system and then reflashing CP 8.2.7. In both cases the CP file system look normal when mounted as CIRCUITPY from linux.

[epeay]

macOS Ventura, v13.6
Ubuntu 22.04.3 LTS, kernel 5.15.0-87-generic

This may or may not be relevant, but when flashing circuitpython yesterday, I found that I couldn't simply drop the file into the RPI-... mount via nautilus. It complained of permissions issues, which factually there were none. I went into the terminal to copy the uf2 file onto the pico w, without needing sudo. Not sure what to make of that.

OK, let's see if I can recreate this. I'm starting from having a working filesystem already. Saved my work.

Running storage.erase_filesystem() appears to have nuked and reset the filesystem. Notably, code.py now contains print("Hello World!"). The first time I got the filesystem working, code.py only contained a string of ÿ characters.

Flashing with micropython v1.21.0. I can drop the uf2 file directly into the mount via nautilus. Ran a small script from code on the pico W filesystem. Everything works.

Flashing with circuitpython 8.2.7. I can drop the file directly into RPI-RPI2 via nautilus.

Comes back just fine. I can see the CIRCUITPYTHON mount. Looks normal. Hrmmm.

===

OK let's mess with the filesystem.

epeay@beast:/media/epeay/CIRCUITPY$ rm code.py; touch some_new_file
epeay@beast:/media/epeay/CIRCUITPY$ ll
total 22
drwxr-xr-x  4 epeay epeay 16384 Dec 31  1969 ./
drwxr-x---+ 6 root  root   4096 Oct 30 00:35 ../
-rw-r--r--  1 epeay epeay   152 Dec 31  2019 boot_out.txt
drwxr-xr-x  2 epeay epeay   512 Dec 31  2019 .fseventsd/
drwxr-xr-x  2 epeay epeay   512 Dec 31  2019 lib/
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 .metadata_never_index
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 settings.toml
-rw-r--r--  1 epeay epeay     0 Oct 30 00:35 some_new_file
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 .Trashes

Reflashing with micropython. Running sample script. Everything's fine.

Reflashing with circuitpython.

OK, here's something. Why is my filesystem being preserved across flashes???

epeay@beast:/media/epeay/CIRCUITPY$ ll
total 22
drwxr-xr-x  4 epeay epeay 16384 Dec 31  1969 ./
drwxr-x---+ 6 root  root   4096 Oct 30 00:39 ../
-rw-r--r--  1 epeay epeay   152 Dec 31  2019 boot_out.txt
drwxr-xr-x  2 epeay epeay   512 Dec 31  2019 .fseventsd/
drwxr-xr-x  2 epeay epeay   512 Dec 31  2019 lib/
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 .metadata_never_index
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 settings.toml
-rw-r--r--  1 epeay epeay     0 Oct 30 00:35 some_new_file
-rw-r--r--  1 epeay epeay     0 Dec 31  2019 .Trashes

Why wouldn't a fresh filesystem be written when flashing circuitpython?

[RetiredWizard]

The boot loader doesn't erase the entire flash unless the size of Circuit Python changes. Performing an ESPTOOL (oops, Nuke UF2, see Dan's update below) flash_erase before flashing can be a good idea when encountering file system problems.

Micro Python apparently places the file system in a different location than Circuit Python. I was surprised by a returning Circuit Python file system I think back as far as version 7.x.

[epeay]

unless the size of Circuit Python changes

How is that defined? I re-flashed 8.2.2, as it is literally a different size than 8.2.7, but I have the same filesystem.

Performing an ESPTOOL flash_erase before flashing can be a good idea when encountering file system problems.

I have bounced back and forth between micropython and circuitpython, with no erasing of storage. This bug seems like circuitpython was trying to make a filesystem out of old micropython data. But without a way to recreate, there's not much else we can do.

General question: If the pico w loads as /dev/sdh1, and I use dd to make an ISO of /dev/sdh (no 1), am I just capturing the user storage, or is that all storage, including the critical bits that facilitate flashing bootloaders and such? I'm guessing the critical bits are on a separate, less volatile storage, but I don't have a lot of microcontroller experience.

I may not have a reason to do this, now that you've told me about flash_erase, but if I wrote all zeros to /dev/sdh (again, no 1), is that a reasonably safe operation, while still preserving the ability to flash bootloaders and such?

[dhalbert]

esptool is for Espressif boards, not RP2040, which I think is what @epeay is using.

For RP2040, use the nuke UF2 to erase all of flash: https://cdn-learn.adafruit.com/assets/assets/000/099/419/original/flash_nuke.uf2

[RetiredWizard]

How is that defined? I re-flashed 8.2.2, as it is literally a different size than 8.2.7, but I have the same filesystem.

I'm really not the best source but my understanding was that Circuit Python reserves an area of flash for the CP code, as long as that area doesn't change size the rest of flash isn't reformatted. The reserved space doesn't appear to change very often.

[tannewt]

The RP2040 has a bootloader in ROM. It isn't in flash and is never loaded.

The UF2 file format specifies the location of each block to write and erases happen as needed, at the smallest unit.

CP has linker scripts and code that define the maximum size of the binary and then place the user file system after that. CP does that here with 1020k as default. Boards can override it. Pico W sets it to 1532k

MP does this inversely by defining how much flash to leave for the user filesystem. On Pico W MICROPY_HW_FLASH_STORAGE_BYTES is set to 848k. That is subtracted from PICO_FLASH_SIZE_BYTES which is 2 * 1024 = 2048k. That means the flash offset is 2048k - 848k = 1200k.

1532k != 1200k so the file systems won't be interchangeable. CP will find the middle of a filesystem from MP. MP may find it's old filesystem just fine because the CP builds may or may not fill (and therefore erase) all the way to the 1532k mark. We like to leave ourselves headroom for the binary size of CP growing.

[epeay]

That is a very thorough response. Thank you. That clears things up quite a bit.

I don't think there's much else to do with this ticket, if the issue can't be recreated. I appreciate your help!