Add : espressif : make spiffs

[itsjustvenky]

In Arduino-esp8266 project there is a option to build and upload just SPIFFS image to ESP.

http://esp8266.github.io/Arduino/versions/2.0.0/doc/filesystem.html#uploading-files-to-file-system

Is it possible to add similar functionality to "PlatformIO" where I create a folder named "data" in the project root folder and set the flag/property to build just SPIFFS image and upload it to ESP using serial or ota

@igrr (and yourself) already provided mentioned the document the option to upload just SPIFFS image to ESP over OTA

https://github.com/esp8266/Arduino/blob/master/doc/platformio.md

Let me know.thanks.

[ivankravets]

@itsjustvenky please provide console output from Arduino IDE of this process. Thanks.

[ivankravets]

@igrr could you explain us how does it work? What we should implement?

Thanks.

[itsjustvenky]

@ivankravets The SPIFFS bin generation and upload to ESP is done a arduino IDE plugin jar file.

https://github.com/esp8266/arduino-esp8266fs-plugin/
http://esp8266.github.io/Arduino/versions/2.0.0/doc/filesystem.html#uploading-files-to-file-system

I am till figuring out the details on the SPIFFS bin file generation.

"igrr" is the correct person to answer your question. thanks.

[ivankravets]

Thanks, I see now how does it work https://github.com/esp8266/arduino-esp8266fs-plugin/blob/master/src/ESP8266FS.java

Will back to this issue as soon as possible.

[itsjustvenky]

I found this too (just reference)

https://github.com/pellepl/spiffs

[itsjustvenky]

Ok this what is the java (Processing app) is using

https://github.com/igrr/mkspiffs

[itsjustvenky]

The binary that is used to generate SPIFFS image is "mkspiffs" and it takes parameters from

(on Win7) C:\Program Files (x86)\Arduino\hardware\esp8266com\esp8266\boards.txt

example

generic.menu.FlashSize.4M1M=4M (1M SPIFFS)
generic.menu.FlashSize.4M1M.build.flash_size=4M
generic.menu.FlashSize.4M1M.build.flash_ld=eagle.flash.4m1m.ld
generic.menu.FlashSize.4M1M.build.spiffs_start=0x300000
generic.menu.FlashSize.4M1M.build.spiffs_end=0x3FB000
generic.menu.FlashSize.4M1M.build.spiffs_blocksize=8192
generic.menu.FlashSize.4M1M.build.spiffs_pagesize=256
generic.menu.FlashSize.4M1M.upload.maximum_size=1044464

this code is from the java file

 if(listenOnProcess(new String[]{toolPath, "-c", dataPath, "-p", spiPage+"", "-b", spiBlock+"", "-s", (spiEnd - spiStart)+"", imagePath}) != 0){

and then the image is getting uploaded into ESP using Esptool

sExec(new String[]{esptoolPath, "-cd", resetMethod, "-cb", uploadSpeed, "-cp", serialPort, "-ca", uploadAddress, "-cf", imagePath});

and the esptool parameters too are taken from "boards.txt" (depending on the board the user selected).

[0x2b3bfa0]

👍

@ivankravets: Practical example of @itsjustvenky post (BASH flavored):

#!/bin/bash

MKSPIFFS="/usr/bin/mkspiffs"
ESPTOOL="/usr/bin/esptool"

PROJECT_FOLDER="/home/ivan/myproject/"
TEMP_FILE=$(mktemp /tmp/spiffs_image-XXXXXXXX.bin)

SERIAL_PORT="/dev/ttyUSB0"
RESET_METHOD="nodemcu"
UPLOAD_SPEED=115200

SPIFFS_START_ADDRESS=0x300000
SPIFFS_END_ADDRESS=0x3FB000
SPIFFS_PAGE=256
SPIFFS_BLOCK=8192
SPIFFS_SIZE=$((SPIFFS_END_ADDRESS - SPIFFS_START_ADDRESS))

$MKSPIFFS -c "${PROJECT_FOLDER}/data" -p "$SPIFFS_PAGE" -b "$SPIFFS_BLOCK" -s "$SPIFFS_SIZE" "$TEMP_FILE"
$ESPTOOL -cd "$RESET_METHOD" -cb "$UPLOAD_SPEED" -cp "$SERIAL_PORT" -ca "$SPIFFS_START_ADDRESS" -cf "$TEMP_FILE"

Note 1: All these uppercase variables must be retrieved from PlatformIO.ini or whatever file that contains the device parameters.
Note 2: I don't know if mkspiffs is distributed along with the esp8266-arduino framework. If not, is hosted here: https://github.com/igrr/mkspiffs

[ivankravets]

Note 1: All these uppercase variables must be retrieved from PlatformIO.ini or whatever file that contains the device parameters.

If user needs to enter all these vars then he will use Make :) PlatformIO should simplify all these things. I'm working on it.

[0x2b3bfa0]

@ivankravets: 😸 Thank you very much! This will be a very great Christmas present for many of us!

[ivankravets]

Guys, let's test it!

• cae5f1f
• 2c4c3a1

Docs http://docs.platformio.org/en/latest/platforms/espressif.html#uploading-files-to-file-system-spiffs

P.S: Need to install the latest development version http://docs.platformio.org/en/latest/installation.html#d-development-version

[0x2b3bfa0]

@ivankravets: Thank you very much! I'll travel now, however I'll try later (in 6h or tomorrow).

This is the better 🎁 for us!

[itsjustvenky]

@ivankravets I will test it sometime tomorrow and get back to you. Thank you.

[itsjustvenky]

Upload to FS seems to writing at wrong address

Uploading 262144 bytes from .pioenvs/esp01_1m/spiffs_image.bin to flash at 0x00000000
erasing flash
size: 040000 address: 000000

The address is correct when building spiffs images, but while writing it is overwriting x0000000

"/home/venkat/.platformio/packages/tool-esptool/esptool" -vv -cd ck -cb 115200 -cp /dev/ttyUSB0 -cf .pioenvs/esp01_1m/spiffs_image.bin -ca 0xbb000

This is from arduino ide

[SPIFFS] data   : G:\Projects\Arduino\Arduino-IDE\test\FSBrowser\data
[SPIFFS] size   : 256
[SPIFFS] page   : 256
[SPIFFS] block  : 4096
/edit.htm.gz
/favicon.ico
/graphs.js.gz
/index.htm
[SPIFFS] upload : C:\Users\VENKAT~1.VEN\AppData\Local\Temp\build5945408886832196661.tmp/FSBrowser.spiffs.bin
[SPIFFS] reset  : ck
[SPIFFS] port   : COM10
[SPIFFS] speed  : 115200
[SPIFFS] address: 0xBB000

Uploading 262144 bytes from C:\Users\VENKAT~1.VEN\AppData\Local\Temp\build5945408886832196661.tmp/FSBrowser.spiffs.bin to flash at 0x000BB000

Note : Please ignore the sizes as the files (spiffs) are different in "arduino ide" project and platformio project.

I used 1M ESP with this LD script : build_flags = -Wl,-Tesp8266.flash.1m256.ld

I am on
PlatformIO, version 2.7.0.dev6

[ivankravets]

@itsjustvenky I've fixed it. Try PlatformIO 2.7.0.dev7

[ivankravets]

@crushedice2000, @comino could you test it?

[0x2b3bfa0]

@ivankravets: It works for me! 👍 Thank you very much!

[georgkreimer]

I'm getting this error here:

▶ platformio run --target uploadfs

[Wed Dec 30 01:27:03 2015] Processing huzzah (platform: espressif, board: huzzah, framework: arduino, build_flags: -Wl,-Tesp8266.flash.4m.ld)

"/Users/georgkreimer/.platformio/packages/tool-mkspiffs/mkspiffs" -c data -p 256 -b 8192 -s 3125248 .pioenvs/huzzah/spiffs_image.bin
/index.html
(["uploadfs"], [".pioenvs/huzzah/spiffs_image.bin"])
Auto-detected UPLOAD_PORT/DISK: /dev/cu.wchusbserial1410
"/Users/georgkreimer/.platformio/packages/tool-esptool/esptool" -vv -cd nodemcu -cb 115200 -cp /dev/cu.wchusbserial1410 -ca 0x100000 -cf .pioenvs/huzzah/spiffs_image.bin
esptool v0.4.6 - (c) 2014 Ch. Klippel ck@atelier-klippel.de
setting board to nodemcu
setting baudrate from 115200 to 115200
setting port from /dev/tty.usbserial to /dev/cu.wchusbserial1410
setting address from 0x00000000 to 0x00100000
espcomm_upload_file
stat .pioenvs/huzzah/spiffs_image.bin success
opening port /dev/cu.wchusbserial1410 at 115200
tcgetattr
tcsetattr
serial open
opening bootloader
resetting board
trying to connect
setting character timeout 0
done
setting character timeout 1
done
espcomm_send_command: sending command header
espcomm_send_command: sending command payload
trying to connect
setting character timeout 0
done
setting character timeout 1
done
scons: *** [uploadfs] Error 255
espcomm_send_command: sending command header
espcomm_send_command: sending command payload
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_open
Uploading 3125248 bytes from .pioenvs/huzzah/spiffs_image.bin to flash at 0x00100000
erasing flash
size: 2fb000 address: 100000
first_sector_index: 256
total_sector_count: 763
head_sector_count: 16
adjusted_sector_count: 747
adjusted_size: 2eb000
espcomm_send_command: sending command header
espcomm_send_command: sending command payload
setting timeout 10000
setting character timeout 100
done
setting timeout 1
setting character timeout 1
done
warning: espcomm_send_command: didn't receive command response
warning: espcomm_send_command(FLASH_DOWNLOAD_BEGIN) failed
closing bootloader
============================================================================= [ ERROR ] Took 10.85 seconds =============================================================================

[ivankravets]

Could you try for test to upload SPIFFS image using Arduino IDE? https://github.com/esp8266/Arduino/blob/master/doc/filesystem.md#flash-layout

Thanks

[georgkreimer]

Looks like that doesn't work either. Hmm, might be my cheap USB UART (CH340) that has only RX, TX, VCC and GND connected. Though uploading the sketch works fine.

Sketch uses 249,920 bytes (23%) of program storage space. Maximum is 1,044,464 bytes.
Global variables use 39,798 bytes (48%) of dynamic memory, leaving 42,122 bytes for local variables. Maximum is 81,920 bytes.
Uploading 254064 bytes from /var/folders/ky/fpk2bmgn4b30zjhyntzgtzhh0000gn/T/build3fadfb4cba027a61b568c284b4c3fb34.tmp/CaptivePortal.ino.bin to flash at 0x00000000
.........................................................................................................................................................................................................................................................
[SPIFFS] data : /Users/georgkreimer/Library/Arduino15/packages/esp8266/hardware/esp8266/2.0.0-rc2/libraries/DNSServer/examples/CaptivePortal/data
[SPIFFS] size : 3052
[SPIFFS] page : 256
[SPIFFS] block : 8192
/index.html
[SPIFFS] upload : /var/folders/ky/fpk2bmgn4b30zjhyntzgtzhh0000gn/T/build3fadfb4cba027a61b568c284b4c3fb34.spiffs/CaptivePortal.spiffs.bin
[SPIFFS] reset : ck
[SPIFFS] port : /dev/cu.wchusbserial1410
[SPIFFS] speed : 115200
[SPIFFS] address: 0x100000

warning: espcomm_sync failed
error: espcomm_open failed
SPIFFS Upload failed!

[itsjustvenky]

ESP-12E basic board (not nodemcu)

like this one

http://i.ebayimg.com/00/s/MTYwMFgxNjAw/z/NL8AAOSwMmBViphp/$_1.JPG

[georgkreimer]

@itsjustvenky AP. I used the OTA-mDNS-SPIFFS.ino example. Using a huzzah module and this for UART All I had taken with me on vacation. I can test with other setups in like 3 days.

[itsjustvenky]

@georgkreimer can you paste the output from esp console (connect to UART) when doing OTA.
If there is reset then we can find what caused the issue.

[georgkreimer]

@itsjustvenky Sure.

�BBH�s���� ��������V������d$��|�l�<�l�c|����;�c�c��gn�dog���c8��lrl;lx�o��$��#g�|d����#��og�l��l �ng$'{���g��l s��'��d�n�

Chip ID: 0xA6046D
Hostname: ESP8266-OTA-a6046d
Wait for WiFi connection.
....................
Can not connect to WiFi station. Go into AP mode.
IP address: 192.168.4.1

See here. Left side platformIO, right side screen on UART

[itsjustvenky]

@georgkreimer There are chances that you might not have 4M flash on your ESP-12 (huzzah)

[georgkreimer]

@itsjustvenky You are probably right. Would make sense since it dies at 34%. I will check how much it really has and report back.

[itsjustvenky]

you can check using this

ESP.getFlashChipSize()

[georgkreimer]

;ld��|�$�|�$�c|ǃ��{�c�c��og�l'o���#x��l{l{$x�o��d��cg�|lČ��c��'o�$��l�'odgs���o��$r��g��l�o�

Chip ID: 0xA6046D
Hostname: ESP8266-OTA-a6046d
FlashChipSize: 4194304

[itsjustvenky]

I should try the OTA in AP mode.

[itsjustvenky]

@georgkreimer see this : esp8266/Arduino#785

[georgkreimer]

Tried ESP.getFlashChipRealSize() but after reboot serial output is just gibberish.

I have another huzzah at home. Will test it when I'm back. Maybe I should open a separate issue when I can confirm it affects huzzah modules. So let's close this here.

[FWeinb]

It seams like data_dir isn't working correctly. Creating the spiffs.bin is working but when trying to upload the file I get:

"/Users/FWeinb/.platformio/packages/tool-mkspiffs/mkspiffs" -c data -p 256 -b 8192 -s 3125248 .pioenvs/nodemcu/spiffs.bin
warning: can't read source directory
scons: *** [.pioenvs/nodemcu/spiffs.bin] Error 1

Using the default folder is working.

[ivankravets]

@FWeinb Could you open separate issue with it? Thanks!

[ivankravets]

@FWeinb did you place data_dir to [platformio] section in platformio.ini?

[FWeinb]

Sorry for the delay. Final exams...
No i didn't know I have to place them there. Sorry for the trouble.

[msurguy]

So, just to be clear, this is the option that needs to be used in order for platformio run --target uploadfs command to work:

[platformio]
data_dir = data_dir

[ivankravets]

@msurguy you can omit this option if the directory is named as data.

[msurguy]

@ivankravets great, thanks for explaining, I was not able to find that in the docs

[ivankravets]

@msurguy we have documented this option in a few places:

• http://docs.platformio.org/en/latest/projectconf.html#data-dir
• http://docs.platformio.org/en/latest/platforms/espressif.html#uploading-files-to-file-system-spiffs

Did you see these pages? Maybe, need to rephrase what do we mean under data_dir?

[msurguy]

@ivankravets exactly, it is not clear from http://docs.platformio.org/en/latest/platforms/espressif.html#uploading-files-to-file-system-spiffs that data_dir in fact should be named as "data"

[ivankravets]

@msurguy thanks! I've just improved doc 9c15acc

Is it better now?

[msurguy]

@ivankravets beautiful! This does it!

[dalbert2]

Sorry if I'm missing something, but the new 2.0.x releases appear to break the upload_port setting in platformio.ini: it looks like it now interprets the upload_port as an IP address instead of a COM port (at least on windows platforms), so for example:

[env:esp12e]
platform = espressif8266
board = esp12e
framework = arduino
upload_port=COM5

Now results in:

Uploading .pioenvs\esp12e\firmware.bin
12:08:00 [DEBUG]: Options: {'timeout': 10, 'esp_ip': 'COM5', 'host_port': 46511, 'image': '.pioenvs\\esp12e\\firmware.bin', 'host_ip': '0.0.0.0', 'auth': '', 'esp_port': 8266, 'spiffs': False, 'debug': True, 'progress': True}
12:08:00 [INFO]: Starting on 0.0.0.0:46511
12:08:00 [INFO]: Upload size: 262160
Sending invitation to COM5 ..........
12:09:41 [ERROR]: No response from the ESP

Any ideas? I didn't see anything helpful in the release notes. How do I tell it to use a specific serial port for upload? (Note: if no upload port is specified, it auto-detects the first serial port, but that's not the one I need to use).

Also suggestion: when creating a new Arduino ESP8266 project, it would be very helpful for new users to include a comment in the auto-created platformio.ini referencing this page:
https://docs.platformio.org/en/latest/platforms/espressif8266.html#configuration

[ivankravets]

Do you have COM5 hostname?

[ivankravets]

@dalbert2 see platformio/platform-espressif8266#134

Please open PlatformIO IDE > PIO Home > Platforms > Updates.