New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add : espressif : make spiffs #382

Closed
itsjustvenky opened this Issue Dec 17, 2015 · 83 comments

Comments

Projects
None yet
7 participants
@itsjustvenky

itsjustvenky commented Dec 17, 2015

In Arduino-esp8266 project there is a option to build and upload just SPIFFS image to ESP.

http://esp8266.github.io/Arduino/versions/2.0.0/doc/filesystem.html#uploading-files-to-file-system

Is it possible to add similar functionality to "PlatformIO" where I create a folder named "data" in the project root folder and set the flag/property to build just SPIFFS image and upload it to ESP using serial or ota

@igrr (and yourself) already provided mentioned the document the option to upload just SPIFFS image to ESP over OTA

https://github.com/esp8266/Arduino/blob/master/doc/platformio.md

Let me know.thanks.

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 17, 2015

@itsjustvenky please provide console output from Arduino IDE of this process. Thanks.

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 17, 2015

@igrr could you explain us how does it work? What we should implement?

Thanks.

@ivankravets ivankravets added the feature label Dec 17, 2015

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 17, 2015

@ivankravets The SPIFFS bin generation and upload to ESP is done a arduino IDE plugin jar file.

https://github.com/esp8266/arduino-esp8266fs-plugin/
http://esp8266.github.io/Arduino/versions/2.0.0/doc/filesystem.html#uploading-files-to-file-system

I am till figuring out the details on the SPIFFS bin file generation.

"igrr" is the correct person to answer your question. thanks.

@ivankravets ivankravets self-assigned this Dec 17, 2015

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 17, 2015

Thanks, I see now how does it work https://github.com/esp8266/arduino-esp8266fs-plugin/blob/master/src/ESP8266FS.java

Will back to this issue as soon as possible.

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 17, 2015

I found this too (just reference)

https://github.com/pellepl/spiffs
@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 17, 2015

Ok this what is the java (Processing app) is using

https://github.com/igrr/mkspiffs
@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 17, 2015

The binary that is used to generate SPIFFS image is "mkspiffs" and it takes parameters from

(on Win7) C:\Program Files (x86)\Arduino\hardware\esp8266com\esp8266\boards.txt

example

generic.menu.FlashSize.4M1M=4M (1M SPIFFS)
generic.menu.FlashSize.4M1M.build.flash_size=4M
generic.menu.FlashSize.4M1M.build.flash_ld=eagle.flash.4m1m.ld
generic.menu.FlashSize.4M1M.build.spiffs_start=0x300000
generic.menu.FlashSize.4M1M.build.spiffs_end=0x3FB000
generic.menu.FlashSize.4M1M.build.spiffs_blocksize=8192
generic.menu.FlashSize.4M1M.build.spiffs_pagesize=256
generic.menu.FlashSize.4M1M.upload.maximum_size=1044464

this code is from the java file

 if(listenOnProcess(new String[]{toolPath, "-c", dataPath, "-p", spiPage+"", "-b", spiBlock+"", "-s", (spiEnd - spiStart)+"", imagePath}) != 0){

and then the image is getting uploaded into ESP using Esptool

sExec(new String[]{esptoolPath, "-cd", resetMethod, "-cb", uploadSpeed, "-cp", serialPort, "-ca", uploadAddress, "-cf", imagePath});

and the esptool parameters too are taken from "boards.txt" (depending on the board the user selected).

@ivankravets ivankravets changed the title from Add : espressif : make spiffs to Add : espressif : make spiffs [$10] Dec 22, 2015

@ivankravets ivankravets added the bounty label Dec 22, 2015

@crushedice2000

This comment has been minimized.

crushedice2000 commented Dec 26, 2015

👍

@ivankravets: Practical example of @itsjustvenky post (BASH flavored):

#!/bin/bash

MKSPIFFS="/usr/bin/mkspiffs"
ESPTOOL="/usr/bin/esptool"

PROJECT_FOLDER="/home/ivan/myproject/"
TEMP_FILE=$(mktemp /tmp/spiffs_image-XXXXXXXX.bin)

SERIAL_PORT="/dev/ttyUSB0"
RESET_METHOD="nodemcu"
UPLOAD_SPEED=115200

SPIFFS_START_ADDRESS=0x300000
SPIFFS_END_ADDRESS=0x3FB000
SPIFFS_PAGE=256
SPIFFS_BLOCK=8192
SPIFFS_SIZE=$((SPIFFS_END_ADDRESS - SPIFFS_START_ADDRESS))

$MKSPIFFS -c "${PROJECT_FOLDER}/data" -p "$SPIFFS_PAGE" -b "$SPIFFS_BLOCK" -s "$SPIFFS_SIZE" "$TEMP_FILE"
$ESPTOOL -cd "$RESET_METHOD" -cb "$UPLOAD_SPEED" -cp "$SERIAL_PORT" -ca "$SPIFFS_START_ADDRESS" -cf "$TEMP_FILE"

Note 1: All these uppercase variables must be retrieved from PlatformIO.ini or whatever file that contains the device parameters.
Note 2: I don't know if mkspiffs is distributed along with the esp8266-arduino framework. If not, is hosted here: https://github.com/igrr/mkspiffs

@ivankravets ivankravets added this to the 2.7.0 milestone Dec 26, 2015

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 27, 2015

Note 1: All these uppercase variables must be retrieved from PlatformIO.ini or whatever file that contains the device parameters.

If user needs to enter all these vars then he will use Make :) PlatformIO should simplify all these things. I'm working on it.

@crushedice2000

This comment has been minimized.

crushedice2000 commented Dec 27, 2015

@ivankravets: 😸 Thank you very much! This will be a very great Christmas present for many of us!

ivankravets added a commit that referenced this issue Dec 27, 2015

ivankravets added a commit that referenced this issue Dec 27, 2015

@ivankravets

This comment has been minimized.

@crushedice2000

This comment has been minimized.

crushedice2000 commented Dec 28, 2015

@ivankravets: Thank you very much! I'll travel now, however I'll try later (in 6h or tomorrow).

This its the better 🎁 for us!

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 28, 2015

@ivankravets I will test it sometime tomorrow and get back to you. Thank you.

ivankravets added a commit that referenced this issue Dec 28, 2015

ivankravets added a commit that referenced this issue Dec 28, 2015

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 29, 2015

Upload to FS seems to writing at wrong address

Uploading 262144 bytes from .pioenvs/esp01_1m/spiffs_image.bin to flash at 0x00000000
erasing flash
size: 040000 address: 000000

The address is correct when building spiffs images, but while writing it is overwriting x0000000

"/home/venkat/.platformio/packages/tool-esptool/esptool" -vv -cd ck -cb 115200 -cp /dev/ttyUSB0 -cf .pioenvs/esp01_1m/spiffs_image.bin -ca 0xbb000

This is from arduino ide

[SPIFFS] data   : G:\Projects\Arduino\Arduino-IDE\test\FSBrowser\data
[SPIFFS] size   : 256
[SPIFFS] page   : 256
[SPIFFS] block  : 4096
/edit.htm.gz
/favicon.ico
/graphs.js.gz
/index.htm
[SPIFFS] upload : C:\Users\VENKAT~1.VEN\AppData\Local\Temp\build5945408886832196661.tmp/FSBrowser.spiffs.bin
[SPIFFS] reset  : ck
[SPIFFS] port   : COM10
[SPIFFS] speed  : 115200
[SPIFFS] address: 0xBB000

Uploading 262144 bytes from C:\Users\VENKAT~1.VEN\AppData\Local\Temp\build5945408886832196661.tmp/FSBrowser.spiffs.bin to flash at 0x000BB000

Note : Please ignore the sizes as the files (spiffs) are different in "arduino ide" project and platformio project.

I used 1M ESP with this LD script : build_flags = -Wl,-Tesp8266.flash.1m256.ld

I am on
PlatformIO, version 2.7.0.dev6

ivankravets added a commit that referenced this issue Dec 29, 2015

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 29, 2015

@itsjustvenky I've fixed it. Try PlatformIO 2.7.0.dev7

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 29, 2015

@crushedice2000, @comino could you test it?

@crushedice2000

This comment has been minimized.

crushedice2000 commented Dec 29, 2015

@ivankravets: It works for me! 👍 Thank you very much!

@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

I'm getting this error here:

▶ platformio run --target uploadfs

[Wed Dec 30 01:27:03 2015] Processing huzzah (platform: espressif, board: huzzah, framework: arduino, build_flags: -Wl,-Tesp8266.flash.4m.ld)

"/Users/georgkreimer/.platformio/packages/tool-mkspiffs/mkspiffs" -c data -p 256 -b 8192 -s 3125248 .pioenvs/huzzah/spiffs_image.bin
/index.html
(["uploadfs"], [".pioenvs/huzzah/spiffs_image.bin"])
Auto-detected UPLOAD_PORT/DISK: /dev/cu.wchusbserial1410
"/Users/georgkreimer/.platformio/packages/tool-esptool/esptool" -vv -cd nodemcu -cb 115200 -cp /dev/cu.wchusbserial1410 -ca 0x100000 -cf .pioenvs/huzzah/spiffs_image.bin
esptool v0.4.6 - (c) 2014 Ch. Klippel ck@atelier-klippel.de
setting board to nodemcu
setting baudrate from 115200 to 115200
setting port from /dev/tty.usbserial to /dev/cu.wchusbserial1410
setting address from 0x00000000 to 0x00100000
espcomm_upload_file
stat .pioenvs/huzzah/spiffs_image.bin success
opening port /dev/cu.wchusbserial1410 at 115200
tcgetattr
tcsetattr
serial open
opening bootloader
resetting board
trying to connect
setting character timeout 0
done
setting character timeout 1
done
espcomm_send_command: sending command header
espcomm_send_command: sending command payload
trying to connect
setting character timeout 0
done
setting character timeout 1
done
scons: *** [uploadfs] Error 255
espcomm_send_command: sending command header
espcomm_send_command: sending command payload
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_send_command: receiving 2 bytes of data
espcomm_open
Uploading 3125248 bytes from .pioenvs/huzzah/spiffs_image.bin to flash at 0x00100000
erasing flash
size: 2fb000 address: 100000
first_sector_index: 256
total_sector_count: 763
head_sector_count: 16
adjusted_sector_count: 747
adjusted_size: 2eb000
espcomm_send_command: sending command header
espcomm_send_command: sending command payload
setting timeout 10000
setting character timeout 100
done
setting timeout 1
setting character timeout 1
done
warning: espcomm_send_command: didn't receive command response
warning: espcomm_send_command(FLASH_DOWNLOAD_BEGIN) failed
closing bootloader
============================================================================= [ ERROR ] Took 10.85 seconds =============================================================================

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 30, 2015

Could you try for test to upload SPIFFS image using Arduino IDE? https://github.com/esp8266/Arduino/blob/master/doc/filesystem.md#flash-layout

Thanks

@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

Looks like that doesn't work either. Hmm, might be my cheap USB UART (CH340) that has only RX, TX, VCC and GND connected. Though uploading the sketch works fine.

Sketch uses 249,920 bytes (23%) of program storage space. Maximum is 1,044,464 bytes.
Global variables use 39,798 bytes (48%) of dynamic memory, leaving 42,122 bytes for local variables. Maximum is 81,920 bytes.
Uploading 254064 bytes from /var/folders/ky/fpk2bmgn4b30zjhyntzgtzhh0000gn/T/build3fadfb4cba027a61b568c284b4c3fb34.tmp/CaptivePortal.ino.bin to flash at 0x00000000
.........................................................................................................................................................................................................................................................
[SPIFFS] data : /Users/georgkreimer/Library/Arduino15/packages/esp8266/hardware/esp8266/2.0.0-rc2/libraries/DNSServer/examples/CaptivePortal/data
[SPIFFS] size : 3052
[SPIFFS] page : 256
[SPIFFS] block : 8192
/index.html
[SPIFFS] upload : /var/folders/ky/fpk2bmgn4b30zjhyntzgtzhh0000gn/T/build3fadfb4cba027a61b568c284b4c3fb34.spiffs/CaptivePortal.spiffs.bin
[SPIFFS] reset : ck
[SPIFFS] port : /dev/cu.wchusbserial1410
[SPIFFS] speed : 115200
[SPIFFS] address: 0x100000

warning: espcomm_sync failed
error: espcomm_open failed
SPIFFS Upload failed!

@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 30, 2015

Maybe set esp8266.flash.4m1m.ld as default LD script? Currently by default is esp8266.flash.4m.ld

@itsjustvenky do you have the same problems with uploading SPIFFS+OTA for 3m?

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

@ivankravets No

"/home/venkat/.platformio/packages/tool-mkspiffs/mkspiffs" -c data -p 256 -b 8192 -s 3125248 .pioenvs/esp12e/spiffs_image.bin
@ivankravets

This comment has been minimized.

Member

ivankravets commented Dec 30, 2015

@itsjustvenky which board did you test? Looks like problem is only with board: huzzah?

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

ESP-12E basic board (not nodemcu)

like this one

http://i.ebayimg.com/00/s/MTYwMFgxNjAw/z/NL8AAOSwMmBViphp/$_1.JPG

@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

@itsjustvenky AP. I used the OTA-mDNS-SPIFFS.ino example. Using a huzzah module and this for UART All I had taken with me on vacation. I can test with other setups in like 3 days.

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

@georgkreimer can you paste the output from esp console (connect to UART) when doing OTA.
If there is reset then we can find what caused the issue.

@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

@itsjustvenky Sure.

�BBH�s���� ��������V������d$��|�l�<�l�c|����;�c�c��gn�dog���c8��lrl;lx�o��$��#g�|d����#��og�l��l �ng$'{���g��l s��'��d�n�

Chip ID: 0xA6046D
Hostname: ESP8266-OTA-a6046d
Wait for WiFi connection.
....................
Can not connect to WiFi station. Go into AP mode.
IP address: 192.168.4.1

See here. Left side platformIO, right side screen on UART

screen shot 2015-12-30 at 5 19 48 pm

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

@georgkreimer There are chances that you might not have 4M flash on your ESP-12 (huzzah)

@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

@itsjustvenky You are probably right. Would make sense since it dies at 34%. I will check how much it really has and report back.

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

you can check using this

ESP.getFlashChipSize()
@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

;ld��|�$�|�$�c|ǃ��{�c�c��og�l'o���#x��l{l{$x�o��d��cg�|lČ��c��'o�$��l�'odgs���o��$r��g��l�o�

Chip ID: 0xA6046D
Hostname: ESP8266-OTA-a6046d
FlashChipSize: 4194304

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

I should try the OTA in AP mode.

@itsjustvenky

This comment has been minimized.

itsjustvenky commented Dec 30, 2015

@georgkreimer

This comment has been minimized.

georgkreimer commented Dec 30, 2015

Tried ESP.getFlashChipRealSize() but after reboot serial output is just gibberish.

I have another huzzah at home. Will test it when I'm back. Maybe I should open a separate issue when I can confirm it affects huzzah modules. So let's close this here.

ivankravets added a commit that referenced this issue Dec 30, 2015

@FWeinb

This comment has been minimized.

FWeinb commented Jan 8, 2016

It seams like data_dir isn't working correctly. Creating the spiffs.bin is working but when trying to upload the file I get:

"/Users/FWeinb/.platformio/packages/tool-mkspiffs/mkspiffs" -c data -p 256 -b 8192 -s 3125248 .pioenvs/nodemcu/spiffs.bin
warning: can't read source directory
scons: *** [.pioenvs/nodemcu/spiffs.bin] Error 1

Using the default folder is working.

@ivankravets

This comment has been minimized.

Member

ivankravets commented Jan 9, 2016

@FWeinb Could you open separate issue with it? Thanks!

@ivankravets

This comment has been minimized.

Member

ivankravets commented Jan 13, 2016

@FWeinb did you place data_dir to [platformio] section in platformio.ini?

@FWeinb

This comment has been minimized.

FWeinb commented Jan 13, 2016

Sorry for the delay. Final exams...
No i didn't know I have to place them there. Sorry for the trouble.

@ivankravets ivankravets changed the title from Add : espressif : make spiffs [$10] to Add : espressif : make spiffs Jan 26, 2016

@ivankravets ivankravets removed the bounty label Jan 26, 2016

@msurguy

This comment has been minimized.

msurguy commented Apr 11, 2016

So, just to be clear, this is the option that needs to be used in order for platformio run --target uploadfs command to work:

[platformio]
data_dir = data_dir
@ivankravets

This comment has been minimized.

Member

ivankravets commented Apr 11, 2016

@msurguy you can omit this option if the directory is named as data.

@msurguy

This comment has been minimized.

msurguy commented Apr 12, 2016

@ivankravets great, thanks for explaining, I was not able to find that in the docs

@ivankravets

This comment has been minimized.

Member

ivankravets commented Apr 12, 2016

@msurguy we have documented this option in a few places:

Did you see these pages? Maybe, need to rephrase what do we mean under data_dir?

@msurguy

This comment has been minimized.

msurguy commented Apr 12, 2016

@ivankravets exactly, it is not clear from http://docs.platformio.org/en/latest/platforms/espressif.html#uploading-files-to-file-system-spiffs that data_dir in fact should be named as "data"

ivankravets added a commit that referenced this issue Apr 13, 2016

@ivankravets

This comment has been minimized.

Member

ivankravets commented Apr 13, 2016

@msurguy thanks! I've just improved doc 9c15acc

Is it better now?

@msurguy

This comment has been minimized.

msurguy commented Apr 19, 2016

@ivankravets beautiful! This does it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment