Need to automate creating and uploading a SPIFFS image from the CLI

[scottchiefbaker]

I'm working on polishing a Makefile for automating compilation and uploading to an ESP8266 board. Using the Arduino IDE I'm able to build and upload a SPIFFS image per these instructions. I am unable to find a way to automate this from the command line.

What commands does the IDE use to compile and upload a SPIFFS image? I found reference to this chunk of code in another issue as a starting point. Is there a way to enable debug mode in the IDE and spit out the exact commands it runs? Once I can figure out the exact commands I can recreate them in my Makefile.

[scottchiefbaker]

I believe I was able to reverse engineer the commands. I'm leaving them here just for searchability if someone else has this issue. Can someone confirm that what I found are the appropriate settings for 1MB, 2MB, and 3MB SPIFFS?

I've integrated it in to my Arduino Makefile when you call make spiffs

Arduino Makefile

1MB Spiffs

mkspiffs -c ~/Arduino/ds18b20/data/ -p 256 -b 8192 -s 1028096 /tmp/out.spiffs
esptool -cd nodemcu -cb 460800 -cp /dev/ttyUSB0 -ca 0x300000 -cf /tmp/out.spiffs

2MB Spiffs

mkspiffs -c ~/Arduino/ds18b20/data/ -p 256 -b 8192 -s 2076672 /tmp/out.spiffs
esptool -cd nodemcu -cb 460800 -cp /dev/ttyUSB0 -ca 0x200000 -cf /tmp/out.spiffs

3MB Spiffs

mkspiffs -c ~/Arduino/ds18b20/data/ -p 256 -b 8192 -s 3125248 /tmp/out.spiffs
esptool -cd nodemcu -cb 460800 -cp /dev/ttyUSB0 -ca 0x100000 -cf /tmp/out.spiffs

[bertrandlo]

@scottchiefbaker i try the command for 1MB but the error message was shown

"error: Image size should be a multiple of block size" 

do you get the same result ??

[scottchiefbaker]

No the commands are working for me:

mkspiffs -c /home/bakers/Arduino/NodeWebServerGZ/data/ --page 256 --block 8192 -s 1028096 /tmp/NodeWebServerGZ.spiffs

Gives me a .spiffs files and no errors.

[canadaduane]

This is my working configuration (Mac OS X Mojave, esp8266 with 2MB filesystem):

(Side note: Has esptool changed to esptool.py and did its arguments change as well? I couldn't find -cd, -cb, -cp, -ca options.)

# Compile https://github.com/igrr/mkspiffs
mkspiffs -c ./firmware/tictactoe/data \
  --size 2072576 \
  --page 256 \
  --block 8192 -- \
  spiffs.bin

# Install via `brew install esptool`
esptool.py --chip esp8266 \
  --port /dev/cu.usbserial-1410 \
  --baud 460800 \
  --before default_reset \
  --after hard_reset \
  write_flash 0x200000 spiffs.bin

# Configured from parameters shown in Arduino IDE:
#
# data    : /Users/duane/Dropbox/tictactoe/firmware/tictactoe/data
# size    : 2024
# page    : 256
# block   : 8192
# /player.txt
# upload  : /var/folders/bl/vmhly6rn6kbd7hphg614m4b80000gn/T/arduino_build_276330/tictactoe.spiffs.bin
# address  : 0x200000
# reset    : --before default_reset --after hard_reset
# port     : /dev/cu.usbserial-1410
# speed    : 921600
# python   : python
# uploader : /Users/duane/Library/Arduino15/packages/esp8266/hardware/esp8266/2.6.1/tools/upload.py

Notes:

• size is 2024kb = 2072576
  • note at Need to automate creating and uploading a SPIFFS image from the CLI #51 says size should be 2076672
  • I'm using 2072576 because = 2024*1024
• although speed is set to 921600, Arduino Makefile says 460800 (see https://www.perturb.org/code/Arduino-Makefile). Using 921600 resulted in errors, A fatal error occurred: Timed out waiting for packet header and A fatal error occurred: Invalid head of packet (0x80)
• the 0x200000 address comes from the fact that I am programming a WeMos ESP8266 and 2MB of its 4MB flash is reserved for filesystem
• I had to remove -arch i386 from mkspiffs Makefile to get it to compile. See Fails to compile in OSX Mojave igrr/mkspiffs#58

[wolfen351]

I also encountered the "error: Image size should be a multiple of block size" error.

I adjusted the 2Mb command like this:

calc 8192*256 2097152

So I tried that number in the command:

~/mkspiffs -c data/ -p 256 -b 8192 -s 2097152 filesystem.spiffs
And it created the filesystem.

[neinseg]

Here is an example of a Makefile that automatically extracts the SPIFFS properties from arduino's build settings:

sketch          := foo.ino
boardconfig     := esp8266:esp8266:generic:eesz=4M1M

ARDUINO_CLI ?= arduino-cli
MKSPIFFS    ?= mkspiffs
BC          ?= bc


all: build

.PHONY: build
build:
        $(ARDUINO_CLI) compile --fqbn $(boardconfig) $(sketch)

.ONESHELL:
filesystem.bin:
        PROPS=$$($(ARDUINO_CLI) compile --fqbn $(boardconfig) --show-properties)
        BUILD_SPIFFS_BLOCKSIZE=$$(echo "$$PROPS"|grep "^build.spiffs_blocksize"|cut -d= -f2)
        BUILD_SPIFFS_END=$$(echo "$$PROPS"|grep "^build.spiffs_end"|cut -d= -f2)
        BUILD_SPIFFS_PAGESIZE=$$(echo "$$PROPS"|grep "^build.spiffs_pagesize"|cut -d= -f2)
        BUILD_SPIFFS_START=$$(echo "$$PROPS"|grep "^build.spiffs_start"|cut -d= -f2)
        BUILD_SPIFFS_SIZE=$$(echo "ibase=16;$${BUILD_SPIFFS_END:2}-$${BUILD_SPIFFS_START:2}"|bc -q)
        echo BUILD_SPIFFS_SIZE $$BUILD_SPIFFS_SIZE
        $(MKSPIFFS) -c data -p $$BUILD_SPIFFS_PAGESIZE -b $$BUILD_SPIFFS_BLOCKSIZE -s $$BUILD_SPIFFS_SIZE $@

.PHONY: flash
flash: filesystem.bin

.PHONY: flash-fs
flash-fs:

.PHONY: clean
        rm -rf build
        rm -f filesystem.bin

[weshouman]

Thanks @neinseg for the Makefile
I've had to do the following modifications to make it work

• Use tabs instead of spaces
• Add clean target
• Update the flash and flash-fs targets

And the following modifications to support ESP32 instead of ESP8266

• blocksize is standardized to 4096
• pagesize is standardized to 256
• spiffs_start and spiffs_size are captured directly from the partition table, for example /root/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/partitions/default.csv

sketch      := foo.ino
CORE        :=esp32:esp32
# Flashing on  an "ESP32 Dev Module" board
boardconfig :="${CORE}:esp32"

ARDUINO_CLI ?= arduino-cli
MKSPIFFS    ?= mkspiffs
BC          ?= bc

PARTITION_TABLE=/root/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/partitions/default.csv

DEVICE :=/dev/ttyUSB0

.PHONY: build
build:
	$(ARDUINO_CLI) compile --fqbn $(boardconfig) $(sketch)

.PHONY: flash
flash:
	$(ARDUINO_CLI) upload -p ${DEVICE} --fqbn ${boardconfig} ${sketch} 

.PHONY: filesystem.bin
.ONESHELL:
filesystem.bin:
	PROPS=$$($(ARDUINO_CLI) compile --fqbn $(boardconfig) --show-properties)
	BUILD_SPIFFS_BLOCKSIZE=4096
	BUILD_SPIFFS_PAGESIZE=256
	BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f4 | xargs)
	BUILD_SPIFFS_START=$$(echo "ibase=16;$${BUILD_SPIFFS_START_HEX:2}"|bc -q)
	echo "BUILD_SPIFFS_START $$BUILD_SPIFFS_START_HEX ($$BUILD_SPIFFS_START)"
	BUILD_SPIFFS_SIZE_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f5 | xargs)
	BUILD_SPIFFS_SIZE=$$(echo "ibase=16;$${BUILD_SPIFFS_SIZE_HEX:2}"|bc -q)
	echo "BUILD_SPIFFS_SIZE  $$BUILD_SPIFFS_SIZE_HEX ($$BUILD_SPIFFS_SIZE)"
	$(MKSPIFFS) -c data -p $$BUILD_SPIFFS_PAGESIZE -b $$BUILD_SPIFFS_BLOCKSIZE -s $$BUILD_SPIFFS_SIZE $@

.PHONY: flash-fs
.ONESHELL:
flash-fs: filesystem.bin
	BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f4 | xargs)
	python /root/.arduino15/packages/esp32/tools/esptool_py/3.0.0/esptool.py --chip esp32 \
	  --port ${DEVICE} \
	  --baud 460800 \
	  --before default_reset \
	  --after hard_reset \
	  write_flash $${BUILD_SPIFFS_START_HEX} filesystem.bin


.PHONY: clean
clean:
	rm -rf build
	rm -f filesystem.bin

[arifainchtein]

Hello,
I am trying to do exactly the same, but i keep getting problems
I have a custom made pcb using Esp32 WROOM D32. If i use the Arduino IDE i can upload the files and everything works fine.

However, when i use the commands to upload and i run the code, i always get the error:

SPIFFS mount failed -10025

My script has two commands. The first one creates the partition:

/home/ari/.arduino15/packages/esp32/tools/mkspiffs/0.2.3/mkspiffs -c /home/ari/Data/DigitalStables/Projects/pancho-tank-flow/data/ --page 256 --block 8192 --size 2686976 /tmp/out.spiffs
The second command uploads the file:

/home/ari/.arduino15/packages/esp32/tools/esptool_py/4.2.1/esptool.py --chip esp32 \ --port /dev/ttyUSB0 \ --baud 921600 \ --before default_reset \ --after hard_reset \ write_flash 0x00290000 /tmp/out.spiffs
The size parameters i got from seeing the arduino ide 1 console as i do the upload.

Any suggestions would be very much appreciated,

[dlarue]

@arifainchtein did you figure out was was missing? I'm trying to use IDE v2.0.3 and just found out there's no SPIFFS upload tool so I too am resorting to commandline spiffs image building and esptool.py uploading but get the same mount failed -10025 error.

[dlarue]

I could not figure out the makefile enough to get it to generate the proper values so I went to bash and ran the commands then hardcoded the makefile variables. From there filesystem.bin and flash-fs both worked with the source compiled in the IDE v2.0.3.

build and flash both ran from the makefile but the result was a crashing upload similar to what I see when the IDE is set to flash freq of 80MHz instead of 40.

My IDE settings for the esp-wroom-32 module is:

CPU Freq=240MHz
Core Debug Level=NONE
Flash Freq=40MHz
Flash Mode=QIO
Flash Size=4MB
Partition Scheme=Default 4MB with spiffs(1.2MB app/1.5MB spiffs)

Makefile:
`sketch := foo.ino
CORE :=esp32:esp32
boardconfig :="${CORE}:esp32"

ARDUINO_CLI ?= arduino-cli
MKSPIFFS ?= mkspiffs
BC ?= bc

PARTITION_TABLE=~/.arduino15/packages/esp32/hardware/esp32/2.0.5/tools/partitions/default.csv

DEVICE :=/dev/ttyUSB0

.PHONY: build
build:
$(ARDUINO_CLI) compile --fqbn $(boardconfig) $(sketch)

.PHONY: flash
flash:
$(ARDUINO_CLI) upload -p ${DEVICE} --fqbn ${boardconfig} ${sketch}

.PHONY: filesystem.bin
.ONESHELL:
filesystem.bin:
PROPS=$$($(ARDUINO_CLI) compile --fqbn $(boardconfig) --show-properties)
BUILD_SPIFFS_BLOCKSIZE=4096
BUILD_SPIFFS_PAGESIZE=256
BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d"," -f4 | xargs)
BUILD_SPIFFS_START_HEX=cat $(PARTITION_TABLE)
BUILD_SPIFFS_START_HEX="0x290000"
BUILD_SPIFFS_START=$$(echo "ibase=16;$${BUILD_SPIFFS_START_HEX:2}"|bc -q)
BUILD_SPIFFS_START=0
echo "BUILD_SPIFFS_START $$BUILD_SPIFFS_START_HEX ($$BUILD_SPIFFS_START)"
BUILD_SPIFFS_SIZE_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f5 | xargs)
BUILD_SPIFFS_SIZE_HEX="0x170000"
BUILD_SPIFFS_SIZE=$$(echo "ibase=16;$${BUILD_SPIFFS_SIZE_HEX:2}"|bc -q)
BUILD_SPIFFS_SIZE=1507328
echo "BUILD_SPIFFS_SIZE $$BUILD_SPIFFS_SIZE_HEX ($$BUILD_SPIFFS_SIZE)"
$(MKSPIFFS) -c data --page $$BUILD_SPIFFS_PAGESIZE --block $$BUILD_SPIFFS_BLOCKSIZE --size $$BUILD_SPIFFS_SIZE $@

.PHONY: flash-fs
.ONESHELL:
flash-fs: filesystem.bin
BUILD_SPIFFS_START_HEX=$$(cat ${PARTITION_TABLE} | grep "^spiffs"|cut -d, -f4 | xargs)
python ~/.arduino15/packages/esp32/tools/esptool_py/4.2.1/esptool.py --chip esp32
--port ${DEVICE}
--baud 460800
--before default_reset
--after hard_reset
write_flash $${BUILD_SPIFFS_START_HEX} filesystem.bin

.PHONY: clean
clean:
rm -rf build
rm -f filesystem.bin `

[arifainchtein]

@dlarue I sort of gave up trying to automate that part of the build and I upload manually using the Arduino IDE 1.8. A bit of a pain but it works. Would love to automate that process and will revisit this from time to time as other priorities allow. Will share here if I have succcess

[dlarue]

I'd posted about this process to an esp32-arduino forum and was told that LittleFS was the supported filesystem now and for about 2 years SPIFFS is in holding pattern. There seems to be some advantage of a real hierarchical filesystem, a bit faster for only a little bit more memory. I've not tried LittleFS yet but might be something to look into.

[ssshake]

Any idea how to do this make file for an esp8266? For example this partition table csv does not exist for the esp8266.

/esp32/hardware/esp32/2.0.5/tools/partitions/default.csv

So I'm not sure how to adapt the filesystem.bin command.

[dlarue]

from what I can figure out, you can create your own default.csv if you set it to the data you need for the esp8266.

From: esp8266/Arduino#2320 (comment)
you see this:
mkspiffs -p 256 -b 8192 -s $((0x3FB000 - 0x300000)) -c path/to/files/dir/ spiffs-image.bin

and since the default.csv looks like this maybe you can modify it? Or do what's mentioned above.
#Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x5000,
otadata, data, ota, 0xe000, 0x2000,
app0, app, ota_0, 0x10000, 0x140000,
app1, app, ota_1, 0x150000,0x140000,
spiffs, data, spiffs, 0x290000,0x170000,

And then there is this discussion and examples:
#51

[Deepakkumar2102]

@canadaduane Hi i am generating same spiffs image as you mention,
but it not working for me .