Save programs on Prime Hub as they get downloaded #75

laurensvalk · 2021-10-08T13:19:22Z

The Prime Hub, Inventor Hub, and Essential Hub have a separate chip to store user programs and other files. This is sometimes referred to as "external" storage, as opposed to the "internal" storage on the STM32 device, on which the firmware is stored.

This PR is a first attempt at:

Accessing the external storage via SPI. Access is blocking. It calls the pbio update loop in between reading/writing chunks in order to keep everything else (more or less) running.
Using littlefs as the file system layer. This uses (the somewhat outdated) V1.7 in order to be compatible with the file system used by the stock firmware of the Prime Hub and Inventor Hub. If we can manage to keep using this, we'd be able to avoid formatting the external storage. This would be nice even if we'd discontinue dual boot support.
Saving main.mpy on external storage when it is downloaded to the hub via Pybricks Code or Pybricksdev. The program can be restarted by pressing the button. This makes the user experience a whole lot nicer, and it is a key feature to making Pybricks useable in First LEGO League. It's part of the #V3.2 roadmap.

There are currently also a few experimental MicroPython functions to read and write files. I've used them for debugging, but I'd like to remove them before merging this. Since file access is blocking, this isn't something we should support in user programs. As long as Bluetooth keeps working well enough, we can probably get away with this for download-and-run since no other critical services like motors are running at that time.

The build size does increase, particularly due to littlefs, getting us quite close to our self-imposed dual boot size limit. So we might want to rethink the dual boot approach at some point. But since we have access to external flash now, there are a few other creative ways we can back up and restore the original firmware, all without DFU. We could also have a closer look at where the boundary really should be; perhaps we're making it harder than we need to.

It is currently not activated on the Essential Hub. It should work with very little modifications, but I just didn't get around to working on this yet. We'll want to check that it uses the same littlefs version and/or file system properties.

Before we can even think about merging this, we'll need to carefully review that we aren't doing any unwanted writes, in order to preserve the integrity of user files and the bootloader partition.

dlech

Nice work. Any reason we couldn't make this non-blocking and put it in pbdrv right away instead of having this temporary implementation?

dlech · 2021-10-15T19:59:47Z

bricks/stm32/main.c

+        if (size > MPY_MAX_BYTES) {
+            return 0;
+        }
+        *buf = m_malloc(size);


instead of copying the file to RAM, it would be nice to just pass the file reader to compiler.

dlech · 2021-10-15T20:00:45Z

bricks/stm32/main.c

 static void stm32_main(void) {
+
+    #if PYBRICKS_HUB_PRIMEHUB
+    mp_hal_delay_ms(500);


what is this delay for?

For some reason Bluetooth wouldn't come up without it.

This initial read/write operation takes fairly long, so maybe there's still a bit too much blocking going on while everything else is initializing.

dlech · 2021-10-15T20:11:18Z

pybricks/util_pb/pb_flash.c

+static bool flash_initialized = false;
+
+// True sets the notCS pin, False resets it
+static void flash_enable(bool enable) {


It there a reason we do this manually instead of setting the pin to AF5 and letting the SPI hardware take care of it automatically?

No reason against this. I've been following various tutorials that did it this way, but it would certainly be nicer if the appropriate drivers can take care of this.

pybricks/util_pb/pb_flash.c

dlech · 2021-10-15T20:22:16Z

pybricks/util_pb/pb_flash.c

+    .erase = block_device_erase,
+    .sync = block_device_sync,
+
+    .read_size = 256,


Does this info come from the data sheet? If so, we should have a comment explaining which chip we have.

No. I think we can read an arbitrary lengths from the device. Since reads are blocking for now, this provides a reasonable balance between read time and giving the run loop enough cycles.

pybricks/util_pb/pb_flash.c

dlech · 2021-10-15T20:27:22Z

pybricks/util_pb/pb_flash.c

+
+    // Close the file
+    if (lfs_file_close(&lfs, &file) != LFS_ERR_OK) {
+        return PBIO_ERROR_FAILED;


Should we map LFS errors to PBIO errors to get a more specific error?

laurensvalk · 2021-10-18T08:31:35Z

Nice work. Any reason we couldn't make this non-blocking and put it in pbdrv right away instead of having this temporary implementation?

No particular reason against it, just lack of skill at this point. Getting this to work seemed like a good learning opportunity and the possibility to try out the resulting user experience (which is positive.)

I made sure to keep everything out of pbdrv and with only minimal changes to stm32/main.c so it would be easy to remove when we get to the proper implementation later.

But now that we've figured out the essentials, maybe doing the right way isn't too bad in terms of time.

laurensvalk · 2021-10-19T10:12:55Z

It there a reason we do this manually instead of setting the pin to AF5 and letting the SPI hardware take care of it automatically?

I didn't get hardware control for NSS to work yet. I tried this with PB12 in GPIO_MODE_AF_PP, GPIO_AF5_SPI2 with `hspi2.Init.NSS = SPI_NSS_HARD_OUTPUT;

Then it reads all zeros after the first read. Some resources [1, 2].

I haven't confirmed with a scope, but if it really does only control the pin during init/deinit, this isn't what we need. We need to drive this pin to start and stop data transfers.

laurensvalk · 2021-10-25T09:47:41Z

I've updated the PR with additional functionality to restore backed up firmware.

First, make sure you are running the official firmware. If you aren't doing so already, do this, for example:

pybricksdev dfu restore ~/Downloads/firmware-backup-v1.3.00.0203-71f6a4.bin

Then check out this PR and follow these steps

poetry install
make -C bricks/primehub deploy

If you don't want to build from source, just get the build for this PR from the CI and run with the latest pybricksdev release:

pybricksdev flash path/to/firmware.zip

This will upload the firmware to the hub and install it, all without DFU.

Then to restore it, connect to the hub using Pybricks Code, and run this:

from pybricks.experimental import restore_firmware

restore_firmware()

This provides some test code to experiment with the file system.

This combines the following commits: pybricks.experimental: Use 4 bytes for flash addresses. The upstream MicroPython board exits 4 byte mode, but 4 bytes are needed beyond 16Mb. pybricks.experimental: Toggle flash to activate write state. pybricks.experimental: Only write enable. Disable happens automatically on each erase and/or write. pybricks.experimental: Drop flash debug print. pybricks.experimental: Wait for write complete.

Make it somewhat more generic so we can experiment with reading/saving a main.py user script from main.c Also enlarge buffers. Also add mkdir command to create /_pybricks if it doesn't exist.

This hub has dedicated storage space for user programs. Now the downloaded program is stored so it can be started with the button later. It is saved to /_pybricks/main.mpy. This needs to be reworked extensively to make file access clean and non-blocking, but it makes the hub much more useable for now.

This makes reading and writing much faster. Also fix lookahead buffer size.

Init and scan file system only once.

This refactors some common code for writing SPI commands and removes some hardcoded parameters.

This adds an experimental function that can restore a previously backed up firmware file.

This is quite useful for hubs that can read and write from external storage. It is currently enabled on all hubs with the PYBRICKS_STM32_OPT_EXTRA_MOD flag. It is also used for the firmware restore process, so we can parse JSON files to check meta data without reinventing the wheel.

Verify the firmware meta data before installing it. Also reset the hub when we are done in order to complete the installation.

Since flash needs to be rewritten anyway, just use #if for hub-specific code.

Since flash read/write is blocking, we shouldn't allow this for user programs. These are useful for debugging the firmare installation process, but we can drop them once we have decent file i/o capability.

This ensures the installer does not give unexpected result when it is used with newer firmware.

laurensvalk · 2022-01-13T15:17:00Z

We've decided to merge this so we can get an early beta version for Prime Hub and Inventor Hub out. 🚀

We'll need to make this non-blocking later.

laurensvalk added the hub: primehub/inventorhub Issues related to the LEGO SPIKE Prime hub and LEGO MINDSTORMS Robot Invetor hub label Oct 8, 2021

dlech reviewed Oct 15, 2021

View reviewed changes

laurensvalk force-pushed the flash branch from 92329ff to b4f5f5b Compare October 25, 2021 09:39

laurensvalk force-pushed the flash branch 2 times, most recently from 9c65f5b to acef7fd Compare October 27, 2021 13:41

laurensvalk force-pushed the master branch from 9450c82 to dcf3891 Compare October 27, 2021 13:44

laurensvalk force-pushed the flash branch from acef7fd to f38cb08 Compare October 27, 2021 13:47

laurensvalk force-pushed the flash branch 5 times, most recently from 0385408 to 15b37eb Compare December 24, 2021 12:09

laurensvalk force-pushed the flash branch from 15b37eb to 06893de Compare January 10, 2022 19:41

laurensvalk added 14 commits January 13, 2022 11:15

pbio/platform/prime_hub: Enable SPI for flash.

55e0701

pybricks.experimental: Read a file from flash.

e1754c0

This provides some test code to experiment with the file system.

pybricks.experimental: Refactor flash code.

5e3f2af

Make it somewhat more generic so we can experiment with reading/saving a main.py user script from main.c Also enlarge buffers. Also add mkdir command to create /_pybricks if it doesn't exist.

pybricks/util_pb/pb_flash: Increase baudrate.

d357484

This makes reading and writing much faster. Also fix lookahead buffer size.

pybricks/util_pb/pb_flash: Move initialization.

0cb0071

Init and scan file system only once.

pybricks/util_pb/pb_flash: Refactor.

8971de7

This refactors some common code for writing SPI commands and removes some hardcoded parameters.

pybricks/util_pb/pb_flash: Add address offset.

b62ffc1

pybricks/util_pb/pb_flash: Add firmware restore.

4b29eb3

This adds an experimental function that can restore a previously backed up firmware file.

pybricks/util_pb/pb_flash: Adds checks to firmware restore.

9c31ae5

Verify the firmware meta data before installing it. Also reset the hub when we are done in order to complete the installation.

pybricks/util_pb: Enable experimental flash on Essential Hub.

8a988ad

Since flash needs to be rewritten anyway, just use #if for hub-specific code.

pybricks.experimental: Hide experimental file i/o.

e75d2c1

Since flash read/write is blocking, we shouldn't allow this for user programs. These are useful for debugging the firmare installation process, but we can drop them once we have decent file i/o capability.

bricks/stm32/install_pybricks: Add compatible firmware versions.

5bf94d9

This ensures the installer does not give unexpected result when it is used with newer firmware.

laurensvalk force-pushed the flash branch from 06893de to 5bf94d9 Compare January 13, 2022 14:47

laurensvalk merged commit f57f312 into master Jan 13, 2022

laurensvalk deleted the flash branch January 13, 2022 15:19

Uh oh!

Save programs on Prime Hub as they get downloaded #75

Save programs on Prime Hub as they get downloaded #75

Uh oh!

Conversation

laurensvalk commented Oct 8, 2021

Uh oh!

dlech left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laurensvalk commented Oct 18, 2021

Uh oh!

laurensvalk commented Oct 19, 2021

Uh oh!

laurensvalk commented Oct 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laurensvalk commented Jan 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

laurensvalk commented Oct 25, 2021 •

edited

Loading