atmel-samd: Allow loading frozen _main.py instead of main.py on flash

[klardotsh]

Freezing modules is already an advanced-ish feature. This takes it a
step further and allow main.py to be frozen as well (as _main.py),
completely removing the "copy files to MSC device" or "use ampy" steps
for workflows that really need it.

This will still fall back to the main.py on flash storage if the frozen
version is not available.

This filename format is basically the same as was used for frozen
boot.py in the ESP8266 port (where __boot.py is used).

There appears to be no regression in my local testing, but I only really
tested the happy paths (nothing frozen at all, or _main.py being
frozen). However, the former of those cases tests the fallback logic, so
I'm pretty sure the "some things are frozen, but _main isn't one of
them" case should work fine.

My local test for this was to add modules/_main.py with the following
contents:

import board
import time
from digitalio import DigitalInOut, Direction, Pull

led = DigitalInOut(board.D13)
led.direction = Direction.OUTPUT

for _ in range(20):
    led.value = not led.value
    time.sleep(0.5)

... and to build the board with make BOARD=itsybitsy_m4_express FROZEN_MPY_DIR=modules, followed by a UF2 flash. I saw a blinking LED
on boot.

[klardotsh]

Totally not bent on this going through as-is or even at all since this was just scratching my own project's itch and the implementation is admittedly a little hacky, but I at least wanted to offer to upstream this bit :) Cheers!

[tannewt]

Thanks! I'd rather have this as a default main.py that isn't frozen per se. Instead it could detect when no main file is available and write a default in that case. That way its easy to edit it to do something else afterwards. Does that still work for you?

Thanks!

[klardotsh]

There's two answers to that: does it work for my project? Not particularly - I needed the ability to flash over UF2/whatever the board in question uses and know for an absolute fact what code would be run on bootup, and that it wouldn't change (and wouldn't be skipped if there was a main.py already on the board somewhere). That said, I can just keep my fork as a fork for that functionality.

Is it something I'd be willing to change this PR up to accomplish? Probably - time's a little crunched this week but I should have time towards the end of the week to give something like that a whirl.

[tannewt]

What are you doing that needs to control exactly what runs each time? Are you using the mass storage device for anything then?

[klardotsh]

In this particular project I'm flashing a mechanical keyboard firmware, and it was both significantly slower and significantly less reliable/stable to flash entrypoints (main.py, in CircuitPython's case) with ampy or with USB MSC (this is also why I'm not thrilled that bossa deploys to M4 devices are currently broken on my system and I'm stuck with UF2, but that's an issue I need to do further debugging on before opening an issue here). If main.py isn't as expected, the user has no keyboard - yikes! Therefore the most stable and sane option for this particular usecase was to flash "all at once" in classic DFU style, rather than flashing firmware and "userspace" separately. It's also worth noting that the entire firmware's source is squashed into FROZEN_MPY_DIR.

Since this is a very consumer-oriented firmware project, I also don't want them to have to think about the deploy process much - plug the keyboard with a CircuitPython-supported microcontroller in, put it in bootloader mode, run the device-specific Makefile target, and go.

You're correct in alluding to my lack of use of MSC - in fact, in my fork of CircuitPython I've outright disabled access to it and the background task that maintains its existence: https://github.com/KMKfw/circuitpython/commit/5c606deecc93c573516c46787486cee846a6d599. Obviously that commit is pretty project-specific and not something I ever intended to mainline :)

[tannewt]

Ah right. Are you planning on advertising that it runs CircuitPython? If so, having the mass storage device is part of what makes CircuitPython CircuitPython.

I'm surprised you had reliability issues with UF2. Its been rock solid for us. We're going to be moving further away from bossa because UF2 is easier.

Are you expecting people to hack their firmware themselves?

[klardotsh]

The project README right now reads "a firmware for (usually mechanical) keyboards, written in MicroPython and CircuitPython", which was my way of subtly tipping a hat to the great work Adafruit has done to make a platform to build this project on. It was a conscious word choice to go with written in rather than, perhaps, runnable with, to make it a bit more clear that while CircuitPython is shipped with only a ~20-some line C diff from upstream (the merge commit you saw is the grand total of changes from upstream right now), you're still flashing the keyboard firmware primarily.

I think it's easier if I link to an example keymap from this project to let you determine how much "hack their firmware themselves" applies. My short and biased answer is "not really, they write something resembling a smart config file". However, this file (and indeed the macro infrastructure in that project entirely) does not preclude doing anything CircuitPython allows - if you want to make a macro that activates an RGB LED strip somewhere, there's not much stopping you, for example.

If referencing "CircuitPython" is a problem due to the frozen main.py and removed MSC deviations, I'll scrub the references and move on, but hopefully clarification was more of what you were asking for than anything?

[tannewt]

Ya, at this point I just want clarification. I understand the desire to have the firmware fixed but I also really like how hackable CircuitPython makes something. (I made a CircuitPython keyboard myself)

I'd also love to have the board defined in the main repo here to make it official and to keep it up to date with us. So, I want to figure out what you need in here to make that work.

A couple ideas come to mind about reliability. First, require a specific key combo to make the mass storage writable to make sure changes are deliberate. Second, safe mode could run the default main.py instead of the edited form on the filesystem. This would allow the keyboard to work when the changes go wrong.

Also, how will people edit the keymap if not over USB mass storage?

Thanks! I'm excited to see this as the first CircuitPython "appliance".

[klardotsh]

If the board you're referring to by the board defined in the main repo here is the MakerDiary NRF MDK, I planned on pull requesting that once I have the port even somewhat stable - I've been busy with a lot of other aspects of this project and left the branch in a pretty ugly WIP state. So far I'm treating it as an effective clone of the feather_nrf52840_express to great success (I don't know about that Feather since it's only been alluded to in source and in Discord, but my initial impression is that it appears to be a largely-same device with a different pinout and no JST). Happy to work with you on mainlining that once I've had time to dive in a bit more.

Those are really interesting ideas, especially the RO flash until unlocked. If I understand correctly, that functionality does not currently exist in CircuitPython? Would be happy to write it (this sounds like an interesting Python-importable library for usb_msc in general), just want to make sure we're speaking the same language.

The defaultable main.py is another really interesting idea - introduces a "unless you're hacking up KMKFW internals, you can edit your keymap on the flash drive, but you can safely trial and error it" I previously hadn't considered.

Right now the keymap is frozen into the full flashed image (hex/uf2/whatever) as frozen module kmk_keyboard_user.py, which is loaded by kmk.entrypoints.global (flashed as _main.py). It's a bit of a convoluted bootup web right now, admittedly, but it was our semi-hackish way of having "absolute state of keyboard guaranteed" builds. Thus, they'd currently be editing a keymap under user_keymaps/**/*.py (Kyle and I have plenty of examples in there with varying levels of functionality examples).

I've opened KMKfw/kmk_firmware#52 to track these ideas and so that we can pull this discussion over to there. I'll go ahead and close this PR for now, since it appears a few better ideas/alternatives have come up. Also, feel free to holler at me on Discord if it's easier for you, klardotsh#0274. @kdb424 is on there as well as kdb424#6738. We're both in the Adafruit Discord as well.