spiffsgen.py output not supported on ESP8266 (IDFGH-4925)

[marcelstoer]

Not sure if this is a bug or a feature request.

Environment

• Development Kit: does not apply
• Kit version (for WroverKit/PicoKit/DevKitC): does not apply
• Module or chip used: does not apply
• IDF version: master
• Build System: Python
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): does not apply
• Operating System: macOS
• Using an IDE?: No
• Power Supply: mains 😉

Problem Description

spiffsgen.py generates SPIFFS images which are not compatible with (Arduino Core 2.7.4) ESP8266. It works fine on ESP32.

Expected Behavior

I work on a cross-platform Python (GUI) app that amongst other stuff generates SPIFFS images to be used on both ESP32 and ESP8266. I was expecting the binary produced by spiffsgen.py to be platform-compatible between ESP8266/ESP32 even though it is hosted in the ESP-IDF project.

Actual Behavior

Writing the generated SPIFFS binary to ESP8266 with esptool is fine (right, why wouldn't it). The ESP8266 app boots fine, opens SPIFFS fine, can write to it, but it does not find any of the content packaged into the .bin.

I hex-diffed the output of the spiffsgen.py and mkspiffs to see if I could make sense of the differences but I failed. There are some minor differences at that beginning and then repeatedly the same diff throughout the file.

Steps to reproduce

1. Build a SPIFFS image: python3 spiffsgen.py 0xFA000 ./app-props my-app.spiffs.bin
2. Flash it at the appropriate address on an ESP8266; in my case WeMos D1 mini (4MB, FS:1MB) esptool.py --port /dev/cu.SLAB_USBtoUART --baud 921600 --before default_reset --after hard_reset write_flash --flash_size detect --flash_mode dio 0x300000 ~/my-app.spiffs.bin
3. Read file from SPIFFS with sketch -> won't find any files. Something like below:

  if (SPIFFS.begin()) {
    Dir dir = SPIFFS.openDir("/");
    while (dir.next()) {
      Serial.print(dir.fileName());
      if(dir.fileSize()) {
        File f = dir.openFile("r");
        Serial.println(f.size());
      }
    }
    File f = SPIFFS.open("/application.properties", "r");
    if (f) {
...

Based on the Git history this may be one for @igrr (as the author of https://github.com/igrr/mkspiffs), @hanxifu or @renzbagaporo

[renzbagaporo]

One thing to consider is that the settings passed to spiffsgen.py should be the same as how SPIFFS is configured in the ESP8266 Arduino core. You can take a look at the arguments passed to spiffsgen.py in ESP-IDF's case here: https://github.com/espressif/esp-idf/blob/master/components/spiffs/project_include.cmake#L33-L44.

You should be able to cross-reference it with https://github.com/arduino/esp8266/blob/master/cores/esp8266/spiffs/spiffs_config.h. I'm not very familiar with the Arduino core codebase, but in ESP-IDF SPIFFS configuration is ultimately in spiffs_config.h.

[marcelstoer]

@renzbagaporo thanks for this constructive feedback! As the image produced by running spiffsgen.py providing only the absolutely required parameters worked perfectly out of the box on the ESP32 I didn't even consider all the extra options. If I got your comment right then spiffsgen.py should be able to produce ESP8266-compatible images - given, the SPIFFS options are in line.

[renzbagaporo]

absolutely required parameters worked perfectly out of the box on the ESP32 I didn't even consider all the extra options

The default arguments in https://github.com/espressif/esp-idf/blob/master/components/spiffs/spiffsgen.py#L460-L510 also have the default values for the respective Kconfig options, i.e. --page-size == CONFIG_SPIFFS_PAGE_SIZE == 256 by default, which is why you experienced this.

spiffsgen.py should be able to produce ESP8266-compatible images - given, the SPIFFS options are in line.

Yup!

[marcelstoer]

Ahhh...my head is about to explode 😦 But I learned a lot in the meantime. So many indirections, sigh.

In the Arduino IDE one uses @me-no-dev's ESP8266FS plugin to upload a SPIFFS image. It's a Java class that

• pulls SPIFFS start, end, page size, block size from the selected board configuration
• invokes @igrr's mkspiffs passing along those parameters

mkspiffs has its own spiffs_config.h that defines all other SPIFFS params at its compile time. Hence, I was looking at the wrong spiffs_config.h (in the Arduino core) for the longest time it seems.

If my analysis above is correct then only the meta length value is different from the spiffsgen.py default (0 vs. 4). As described the block size depends on the selected board and in my case I need to overwrite your default with 8192.

Still, no dice so far. I continue debugging and looking at hex diffs 😭

[igrr]

The other difference is SPIFFS_ALIGNED_OBJECT_INDEX_TABLES which is 1 in arduino-esp8266, and can't be changed there for compatibility reasons (see the build script; although this is not the build script which the recent releases of Arduino use, it is still compatible with arduino-esp8266).
I'm afraid that support for index table alignment is not implemented in spiffgen.py.

Edit: also SPIFFS_USE_MAGIC_LENGTH=0 in arduino-esp8266, can be adjusted using --use-magic-len flag of spiffsgen.py.

[marcelstoer]

Thanks for the pointer to your build script; one more place to look 😉 I also stumbled upon SPIFFS_USE_MAGIC_LENGTH=0 but didn't figure out how turn this off with spiffsgen.py as it's a boolean param enabled by default (can't just say --use-magic-len 0 or --use-magic-len false).

I'm afraid that support for index table alignment is not implemented in spiffgen.py.

I was afraid I'd sooner or later hit a road block in my attempt to build something cross-platform for both ESP32/ESP8266 😞 Relying on Python is usually a good idea.

[igrr]

On --use-magic-len not being possible to turn off — that's definitely something that can be fixed. I'm not sure how much more difficult would it be to add support for ALIGNED_OBJECT_INDEX_TABLES. From the name it sounds rather simple (something needs to either be aligned or not?), most work probably is to go through spiffs source code and understand what and how exactly needs to be aligned.

[DeeKey]

We also run across this problem while trying to add the feature to flash user configuration (mainly the WiFi credentials). It does not support ESP8266 right now.
And we are currently not able to find a working solution in Python.
Hope that you will solve this problem soon!

If you are curious - this is our work in progress:
https://github.com/pjgueno/customFlasher

[marcelstoer]

Thanks for chiming in. I rely on spiffsgen.py to build a desktop app store (self-contained binary) https://twitter.com/thingpulse/status/1367602216127848448 Looks like your use case is very similar. It would be a real pity if we couldn't extend it to support the ESP8266-based devices.

[igrr]

I'm sorry, given that the esp8266/Arduino project is deprecating spiffs in favor of littlefs, I doubt that supporting esp8266 compatibility in spiffsgen.py will be considered very important.
That said, we can definitely accept PRs to add esp8266 compatibility in spiffsgen.py, should you get it working.

[igrr]

@marcelstoer please try the attached patch to spiffsgen.py and let me know if it solves your problem. You need to specify additional arguments to generate an ESP8266 compatible image. For example, given the values from 16M/15M config you have linked to:

$IDF_PATH/components/spiffs/spiffsgen.py --page-size 256 --block-size 8192 --no-magic-len --aligned-obj-ix-tables --meta-len=0 $((0xFFA000-0x100000)) path/to/files image.bin

6717.patch.zip

[DeeKey]

I was able to write and read the file which was prepared with patched spiffsgen.py
The only note is that in the above mentioned setup it was: WeMos D1 mini (4MB, FS:1MB)
So seems that above mentioned offset which is for 16M/15M might not work... Just use the one for your setup!

[marcelstoer]

@igrr thank you so much! Finally got around to testing this for real. First test was a break-through 🎉 Looking good so far 👍

Will do some more testing over the weekend.

[marcelstoer]

$ ./spiffsgen.py -h produces misleading output for the boolean flags:

  --use-magic           Use magic number to create an identifiable SPIFFS image. Specify if CONFIG_SPIFFS_USE_MAGIC. (default: True)
  --no-magic            Inverse of --use-magic (default: True)
  --use-magic-len       Use position in memory to create different magic numbers for each block. Specify if CONFIG_SPIFFS_USE_MAGIC_LENGTH. (default: True)
  --no-magic-len        Inverse of --use-magic-len (default: True)

-> all are reportedly True by default