Finish off UF2 build support for MP at least for the ESP32-S2/S3 port

[UnexpectedMaker]

@dpgeorge @mattytrentini Can we please finish off the work needed to get UF2 builds out of MP?
From my understanding, it's just the renaming of the FS partition so that the UF2 bootloader can find it is all that's needed?

Mat already got the uf2 build part done.

I feel this is REALLY important to make happen and quickly.

Thanks!

[dpgeorge]

The UF2 bootloader puts some constraints on the flash layout, how to enter the bootloader, and how to flash using the standard idf.py flashing command. So I don't think all ESP32 boards should be changed to support UF2. I propose instead that it be an option for a given board to configure, eg UM_FEATHERS2 can enable it.

[dpgeorge]

Link to relevant docs for the UF2 bootloader: https://github.com/adafruit/tinyuf2/tree/master/ports/espressif

[UnexpectedMaker]

Hmm, ok, I feel this is a big opportunity missed if we can't find a way to allow folks to switch between CircuitPython, MicroPython, Arduino and whatever else that want to use that supports UF2 as easily as just going into the UF2 bootloader and loading the required firmware.uf2.

Not everyone sticks to 1 dev platform, and most people new to Microcontrollers and specifically the ESP32-S range tend to get put off by the whole "esptool flashing MP and then having to reload UF2 bootloader and and then CPY or Arduino" to get things back.

It puts MP into a "not going to try" category because of the that barrier, and we want no barriers to folks "trying" MicroPython.

What about at least having the MP firmware respect the UF2 boot loader and not stomp it when flashing MP, so folks can get back into it if/when they want?

[dpgeorge]

What about at least having the MP firmware respect the UF2 boot loader and not stomp it when flashing MP, so folks can get back into it if/when they want?

Because the UF2 bootloader sits right in the middle of flash and takes up 256k. Projects not wanting to use UF2 (which is everything up until now) are therefore going to waste flash, and also have to reorganise their flash/filesystem layout.

Furthermore, if the UF2 bootloader decides that it needs to change location/size, then we also need to update to follow suit, and so do all esp32-MicroPython projects.

What boards currently ship with the UF2 bootloader installed? It would make sense to support those with UF2.

If UF2 is going to be really popular then we could consider providing two builds of all boards: standard, and UF2 compatible.

[UnexpectedMaker]

Right now all of my S2 and S3 boards ship with CPY and UF2 bootloader. So I'd like UF2 options for them. I'd assume if I had them for my boards, many other folks that use "other" ESP32-S2/S3 boards will want to get it as well... like the S2/S3 dev-kits for example. Just seeing the number of boards that get added to the CPY list as an indicator UF2 is super popular - I'd assume questions of "why is this not available for XXX board" might become common.

[dpgeorge]

If a board does not ship with the UF2 bootloader and we only have firmware that relies on it being there, then that won't work, it will confuse users that they need to install a bootloader first.

Just seeing the number of boards that get added to the CPY list as an indicator UF2 is super popular

If it's added to the CircuitPython list, does that mean it ships with UF2 out of the factory? Or just that CircuitPython now supports it and provides instructions on how to install the UF2 bootloader first?

[UnexpectedMaker]

It totally depends. An S3 devkit from Espressif does not ship with a UF2 bootloader on it. But Adafruit supply one for it and CPY to go with it.

Supplying a MP UF2 version for boards Adafruit have CPY for is totally safe because Adafruit supply UF2 boot loaders for them as well.
If you look at my FeatherS3 board listing at the bottom.
https://circuitpython.org/board/unexpectedmaker_feathers2/

[dpgeorge]

Ok, I think I might have been on the wrong track with this. It might be relatively easy to provide .uf2 files for all boards without any change.

[dpgeorge]

I have this working now. The changes needed are:

• support "ffat" and "vfs" as the name of the partition for the filesystem
• update to the latest uf2conv.py script
• add a step in the esp32 build process that converts micropython.bin to micropython.uf2 using the above script
• make micropython.uf2 available alongside other firmware files on the download page, with instructions how to flash it

[UnexpectedMaker]

I have this working now. The changes needed are:

• support "ffat" and "vfs" as the name of the partition for the filesystem
• update to the latest uf2conv.py script
• add a step in the esp32 build process that converts micropython.bin to micropython.uf2 using the above script
• make micropython.uf2 available alongside other firmware files on the download page, with instructions how to flash it

So a PR and Merge today? :D
Great job @dpgeorge !

[dpgeorge]

See #8246.

[UnexpectedMaker]

That's wicked! Thanks for getting onto this so quickly!
Gonna close this :)

[fpgirard]

I totally agree with UnexpectedMaker's assessment of uf2's value for micropython. We've been using uP for commercial projects for several years now and love it. We're now in a situation where we're developing products that will need to be configured by less technical people and having them drag/drop configuration files using uf2 will be really valuable. This evening, I tried to get uf2 running on the latest nightly build without success. Here's what I tried:

1. Since I'm using a Lolin/Wemos esp32s2 mini, I grabbed Adafruit's tinyuf2 build for the Lolin ESP32S2 mini ( tinyuf2-lolin_s2_mini-0.8.0.zip ) from here.
2. After unzipping it, I successfully flashed it using:
   $ esptool.py --chip esp32s2 --port /dev/ttyACM0 erase_flash
$ esptool.py --chip esp32s2 -p /dev/ttyACM0 write_flash 0x0 combined.bin
3. Resetting the esp, the device mounted as S2MINIBOOT and I dragged over adafruit-circuitpython-lolin_s2_mini-en_US-7.1.1.uf2 file to the mounted device. Upon reset, the device mounts as CIRCUITPY and runs circuitpython.
4. I've repeated the exact sequence but copied over LOLIN_S2_MINI-20220218-unstable-v1.18-141-gff9c70850.uf2. The file does succeed (I think?) in copying, however, the drive fails to mount on reset.
   I'm not really sure where to even begin debugging at this point but will take advice/instructions to help get this important feature working.

[UnexpectedMaker]

I totally agree with UnexpectedMaker's assessment of uf2's value for micropython. We've been using uP for commercial projects for several years now and love it. We're now in a situation where we're developing products that will need to be configured by less technical people and having them drag/drop configuration files using uf2 will be really valuable. This evening, I tried to get uf2 running on the latest nightly build without success. Here's what I tried:

1. Since I'm using a Lolin/Wemos esp32s2 mini, I grabbed Adafruit's tinyuf2 build for the Lolin ESP32S2 mini ( tinyuf2-lolin_s2_mini-0.8.0.zip ) from here.
2. After unzipping it, I successfully flashed it using:
   $ esptool.py --chip esp32s2 --port /dev/ttyACM0 erase_flash
$ esptool.py --chip esp32s2 -p /dev/ttyACM0 write_flash 0x0 combined.bin
3. Resetting the esp, the device mounted as S2MINIBOOT and I dragged over adafruit-circuitpython-lolin_s2_mini-en_US-7.1.1.uf2 file to the mounted device. Upon reset, the device mounts as CIRCUITPY and runs circuitpython.
4. I've repeated the exact sequence but copied over LOLIN_S2_MINI-20220218-unstable-v1.18-141-gff9c70850.uf2. The file does succeed (I think?) in copying, however, the drive fails to mount on reset.
   I'm not really sure where to even begin debugging at this point but will take advice/instructions to help get this important feature working.

You've confused what MicroPython on UF2 brings us... it has nothing to do with USB mass storage.

A UF2 build of MicroPython just means that, assuming you have a board wit the UF2 bootloader on it, you can flash your board with MicroPython without using esptool (on the ESP32) and it wont stomp the UF2 bootloader, so you can go back into the UF2 bootloader and copy CircuitPython back on.

It's about moving between dev environments and not requiring putting the UF2 back on after flashing MP like what happens now.

MicroPython doesn't offer mounting as a USB mass storage device, and this doesn't change that. You still only get a USB serial device like you've always had.

[fpgirard]

Thanks @UnexpectedMaker for the clarification. I'm actually trying (and failing) to reproduce what you've explained. I'm successful in creating a UF2 device with what I believe is the correct UF2 bootloader bin file from Adafruit. On this device, I can drop a circuitpython uf2 file successfully, however, I cannot replicate this behavior (call it 'dev environment change') with the latest nightly micropython uf2 file. The latter, while it does install micropython, stomps my preexisting uf2 bootloader thus disabling my ability to switch back to circuitpython.

Perhaps the micropython forum is a better venue for me to ask for help or advice?

Lastly, I will say that circuitpython's ability to share a file system on a host OS is very appealing.

[ThinkTransit]

Are there any special flags that need to be set to build the UF2 file?

When I run:
idf.py -D MICROPY_BOARD=UM_FEATHERS2 build
It only builds micropython.bin, no micropython.uf2 in build folder

I have tried putting print statements in makeimg.py and it seems like it isn't being called.

idf is v4.4.1 and micropython is latest master branch

Thanks