New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPIFFS support #411

Closed
tijnkooijmans opened this Issue Jun 1, 2017 · 41 comments

Comments

Projects
None yet
@tijnkooijmans

tijnkooijmans commented Jun 1, 2017

Could you share the status of SPIFFS support? In migrating our ESP8266 projects to ESP32 this is the one thing that is currently holding us up. Looking forward to an eta update.

All the best,
Tijn

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Jun 1, 2017

Member

After I'm done with my current assignment, next is SPIFFS. FS is already implemented and SPIFFS will extend it the same way that SD and SD_MMC currently do, so you can maybe start from that and switch to SPIFFS when done.

Member

me-no-dev commented Jun 1, 2017

After I'm done with my current assignment, next is SPIFFS. FS is already implemented and SPIFFS will extend it the same way that SD and SD_MMC currently do, so you can maybe start from that and switch to SPIFFS when done.

@tijnkooijmans

This comment has been minimized.

Show comment
Hide comment
@tijnkooijmans

tijnkooijmans Jun 1, 2017

Thanks for that fast response! I tried the SD library and it works well on my dev board, but unfortunately the production HW doesn't allow for an SD card.

tijnkooijmans commented Jun 1, 2017

Thanks for that fast response! I tried the SD library and it works well on my dev board, but unfortunately the production HW doesn't allow for an SD card.

@bestpika

This comment has been minimized.

Show comment
Hide comment
@bestpika

bestpika Jun 9, 2017

I need SPIFFS to save config.

bestpika commented Jun 9, 2017

I need SPIFFS to save config.

@bbx10

This comment has been minimized.

Show comment
Hide comment
@bbx10

bbx10 Jun 9, 2017

Contributor

@bestpika Take a look at the Preferences library. I used it to save config info.

Contributor

bbx10 commented Jun 9, 2017

@bestpika Take a look at the Preferences library. I used it to save config info.

@bestpika

This comment has been minimized.

Show comment
Hide comment
@bestpika

bestpika Jun 9, 2017

@bbx10 Can the library save array?

Oh, I can save json string and use ArduinoJson to parse it.

bestpika commented Jun 9, 2017

@bbx10 Can the library save array?

Oh, I can save json string and use ArduinoJson to parse it.

@gmag11

This comment has been minimized.

Show comment
Hide comment
@gmag11

gmag11 commented Jun 14, 2017

@Buffalchill

This comment has been minimized.

Show comment
Hide comment
@Buffalchill

Buffalchill Jun 22, 2017

realy need the SPIFFS on the ESP32 :-( SD works but its no option because to much hardware

Buffalchill commented Jun 22, 2017

realy need the SPIFFS on the ESP32 :-( SD works but its no option because to much hardware

@copercini

This comment has been minimized.

Show comment
Hide comment
@copercini

copercini Jul 6, 2017

Collaborator

Maybe this help until the official one https://github.com/copercini/arduino-esp32-SPIFFS

Collaborator

copercini commented Jul 6, 2017

Maybe this help until the official one https://github.com/copercini/arduino-esp32-SPIFFS

@lonerzzz

This comment has been minimized.

Show comment
Hide comment
@lonerzzz

lonerzzz Jul 14, 2017

Contributor

@copercini captured your reference under projects

Contributor

lonerzzz commented Jul 14, 2017

@copercini captured your reference under projects

@lonerzzz

This comment has been minimized.

Show comment
Hide comment
@lonerzzz

lonerzzz Jul 14, 2017

Contributor

@me-no-dev Is the SPIFFs work something that you are actively working? Do you need assistance, for development or testing or otherwise? I will try to capture the current state under projects if you have details.

Contributor

lonerzzz commented Jul 14, 2017

@me-no-dev Is the SPIFFs work something that you are actively working? Do you need assistance, for development or testing or otherwise? I will try to capture the current state under projects if you have details.

@lonerzzz

This comment has been minimized.

Show comment
Hide comment
@lonerzzz

lonerzzz Jul 15, 2017

Contributor

@copercini Can you elaborate on the "Known Issue" regarding "It doesn't support file upload yet." Are you referring to the ESP8266 Sketch Data Upload tool not supporting ESP32 or something else? Are you working to resolve the issue?

Contributor

lonerzzz commented Jul 15, 2017

@copercini Can you elaborate on the "Known Issue" regarding "It doesn't support file upload yet." Are you referring to the ESP8266 Sketch Data Upload tool not supporting ESP32 or something else? Are you working to resolve the issue?

@copercini

This comment has been minimized.

Show comment
Hide comment
@copercini

copercini Jul 15, 2017

Collaborator

@lonerzzz

Are you referring to the ESP8266 Sketch Data Upload tool not supporting ESP32 or something else?

Exactly!
I'm without my board now, but next week I will take a look on this =)

Collaborator

copercini commented Jul 15, 2017

@lonerzzz

Are you referring to the ESP8266 Sketch Data Upload tool not supporting ESP32 or something else?

Exactly!
I'm without my board now, but next week I will take a look on this =)

@copercini

This comment has been minimized.

Show comment
Hide comment
@copercini

copercini Jul 17, 2017

Collaborator

Looks like it’s just add another "if" for ESP32 here which parses partitions CSV

And recompile mkspiffs with ESP32 headers from here

But @me-no-dev started the official implementation, so it’s best wait he definitive solution.

Collaborator

copercini commented Jul 17, 2017

Looks like it’s just add another "if" for ESP32 here which parses partitions CSV

And recompile mkspiffs with ESP32 headers from here

But @me-no-dev started the official implementation, so it’s best wait he definitive solution.

@pbecchi

This comment has been minimized.

Show comment
Hide comment
@pbecchi

pbecchi Jul 23, 2017

Contributor

If "official implementation is started" do you have one expected release date?
If I do start using Copercini branch I will have to change my code later?

Thanks!

Contributor

pbecchi commented Jul 23, 2017

If "official implementation is started" do you have one expected release date?
If I do start using Copercini branch I will have to change my code later?

Thanks!

@bbx10

This comment has been minimized.

Show comment
Hide comment
@bbx10

bbx10 Jul 26, 2017

Contributor

An alternative to the Sketch Data Upload tool is to upload the files using HTTP POST. For example, load the FSBrowser example, then

$ curl -X POST -F "data=@index.htm"     http://<ESP32 IP address>/edit >/dev/null
$ curl -X POST -F "data=@graphs.js.gz"  http://<ESP32 IP address>/edit >/dev/null
$ curl -X POST -F "data=@favicon.ico"   http://<ESP32 IP address>/edit >/dev/null
$ curl -X POST -F "data=@edit.htm.gz"   http://<ESP32 IP address>/edit >/dev/null
Contributor

bbx10 commented Jul 26, 2017

An alternative to the Sketch Data Upload tool is to upload the files using HTTP POST. For example, load the FSBrowser example, then

$ curl -X POST -F "data=@index.htm"     http://<ESP32 IP address>/edit >/dev/null
$ curl -X POST -F "data=@graphs.js.gz"  http://<ESP32 IP address>/edit >/dev/null
$ curl -X POST -F "data=@favicon.ico"   http://<ESP32 IP address>/edit >/dev/null
$ curl -X POST -F "data=@edit.htm.gz"   http://<ESP32 IP address>/edit >/dev/null
@boneskull

This comment has been minimized.

Show comment
Hide comment
@boneskull

boneskull Aug 7, 2017

great success w/ @bbx10's FSBrowser example (here) and @copercini's SPIFFS implementation

boneskull commented Aug 7, 2017

great success w/ @bbx10's FSBrowser example (here) and @copercini's SPIFFS implementation

@copercini

This comment has been minimized.

Show comment
Hide comment
@copercini

copercini Aug 7, 2017

Collaborator

@pbecchi It's implemented like me-no-dev comment above

FS is already implemented and SPIFFS will extend it the same way that SD and SD_MMC currently do, so you can maybe start from that and switch to SPIFFS when done.

So I think you will not need to change some or much code


@boneskull nice to hear it =)

Collaborator

copercini commented Aug 7, 2017

@pbecchi It's implemented like me-no-dev comment above

FS is already implemented and SPIFFS will extend it the same way that SD and SD_MMC currently do, so you can maybe start from that and switch to SPIFFS when done.

So I think you will not need to change some or much code


@boneskull nice to hear it =)

@pbecchi

This comment has been minimized.

Show comment
Hide comment
@pbecchi

pbecchi Aug 7, 2017

Contributor

Thanks
I already started working with Spiffs! It works perfect!

Contributor

pbecchi commented Aug 7, 2017

Thanks
I already started working with Spiffs! It works perfect!

@lonerzzz

This comment has been minimized.

Show comment
Hide comment
@lonerzzz

lonerzzz Aug 18, 2017

Contributor

@copercini Where do we stand with the implementation of the Sketch Data Upload tool for ESP32? Did you start this, is it deferred or in some other state. I ask because if there is no plan for this or you are busied out with all your other contributions, it is something I could work on. Just let me know any status details.
@me-no-dev Is this part of the work you are planning on SPIFFS?

Contributor

lonerzzz commented Aug 18, 2017

@copercini Where do we stand with the implementation of the Sketch Data Upload tool for ESP32? Did you start this, is it deferred or in some other state. I ask because if there is no plan for this or you are busied out with all your other contributions, it is something I could work on. Just let me know any status details.
@me-no-dev Is this part of the work you are planning on SPIFFS?

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Aug 18, 2017

Member

It is planned but depends on mkspiffs and it being on the same spiffs version ;) will get it done ;) ota already supports it

Member

me-no-dev commented Aug 18, 2017

It is planned but depends on mkspiffs and it being on the same spiffs version ;) will get it done ;) ota already supports it

@copercini

This comment has been minimized.

Show comment
Hide comment
@copercini

copercini Aug 18, 2017

Collaborator

As I understand @me-no-dev will re-write SPIFFS things, so unofficial development is paused, but pretty stable =)

Will talk with @me-no-dev about mkspiffs, because I have something working here...

Collaborator

copercini commented Aug 18, 2017

As I understand @me-no-dev will re-write SPIFFS things, so unofficial development is paused, but pretty stable =)

Will talk with @me-no-dev about mkspiffs, because I have something working here...

@tijnkooijmans

This comment has been minimized.

Show comment
Hide comment
@tijnkooijmans

tijnkooijmans Sep 11, 2017

Great, I'm using your lib now @copercini for the time being and it works well.

tijnkooijmans commented Sep 11, 2017

Great, I'm using your lib now @copercini for the time being and it works well.

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Sep 12, 2017

Member

SPIFFS is in ;)

Member

me-no-dev commented Sep 12, 2017

SPIFFS is in ;)

@me-no-dev me-no-dev closed this Sep 12, 2017

@PhilippRott

This comment has been minimized.

Show comment
Hide comment
@PhilippRott

PhilippRott Sep 12, 2017

Thanks me-no-dev for your work on SPIFFS!
Tried your example code with following results:
Hardware: HIMALAYA-ESP32-Dev-Board
Arduino 1.8.2

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0010,len:4
load:0x3fff0014,len:732
load:0x40078000,len:0
load:0x40078000,len:11572
entry 0x40078a14
E (180) SPIFFS: mount failed, -10025
SPIFFS Mount Failed

Any recommendations to fix this problem?

PhilippRott commented Sep 12, 2017

Thanks me-no-dev for your work on SPIFFS!
Tried your example code with following results:
Hardware: HIMALAYA-ESP32-Dev-Board
Arduino 1.8.2

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0010,len:4
load:0x3fff0014,len:732
load:0x40078000,len:0
load:0x40078000,len:11572
entry 0x40078a14
E (180) SPIFFS: mount failed, -10025
SPIFFS Mount Failed

Any recommendations to fix this problem?

@bbx10

This comment has been minimized.

Show comment
Hide comment
@bbx10

bbx10 Sep 12, 2017

Contributor

@PhilippRott @me-no-dev Try adding true as the first parameter to begin(). This fixed the problem for me. The first parameter is formatOnFail so SPIFFS is formatted if the mount fails.

SPIFFS.begin(true);

@copercini Thanks for the unofficial SPIFFS library. It has worked without fail since you released it.

Contributor

bbx10 commented Sep 12, 2017

@PhilippRott @me-no-dev Try adding true as the first parameter to begin(). This fixed the problem for me. The first parameter is formatOnFail so SPIFFS is formatted if the mount fails.

SPIFFS.begin(true);

@copercini Thanks for the unofficial SPIFFS library. It has worked without fail since you released it.

@copercini

This comment has been minimized.

Show comment
Hide comment
@copercini

copercini Sep 13, 2017

Collaborator

@bbx10 Thanks! you just made my day =)

Collaborator

copercini commented Sep 13, 2017

@bbx10 Thanks! you just made my day =)

@PhilippRott

This comment has been minimized.

Show comment
Hide comment
@PhilippRott

PhilippRott Sep 13, 2017

@bbx10 Thanks, worked for me perfect!

PhilippRott commented Sep 13, 2017

@bbx10 Thanks, worked for me perfect!

@trycoon

This comment has been minimized.

Show comment
Hide comment
@trycoon

trycoon Sep 14, 2017

Is Preference (https://github.com/espressif/arduino-esp32/blob/master/libraries/Preferences/src/Preferences.h) reading and writing to the same memory region as SPIFFS? Is there a memory map somewhere showing the different memory regions and there default sizes for the ESP32?

trycoon commented Sep 14, 2017

Is Preference (https://github.com/espressif/arduino-esp32/blob/master/libraries/Preferences/src/Preferences.h) reading and writing to the same memory region as SPIFFS? Is there a memory map somewhere showing the different memory regions and there default sizes for the ESP32?

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Sep 14, 2017

Member

No it does not. There is partition scheme https://github.com/espressif/arduino-esp32/blob/master/tools/partitions/default.csv (Preferences go to NVS)

Member

me-no-dev commented Sep 14, 2017

No it does not. There is partition scheme https://github.com/espressif/arduino-esp32/blob/master/tools/partitions/default.csv (Preferences go to NVS)

@trycoon

This comment has been minimized.

Show comment
Hide comment
@trycoon

trycoon Sep 14, 2017

@me-no-dev Thanks! Looking at the partition scheme, it seems like the max program size is 81920 bytes (0x140000) since that whats fits into the ota-space. So my application is located in "ota_0" and when I update the program using OTA it will be uploaded to "ota_1" and then a pointer is changes to point to "ota_1" and the device is restarted? Next time it will upload to "ota_0" and re-point to ota_0, is that how it works?
Adding up the memory-spaces gives me 81920*2+8192+20480+4096+93952=290560 bytes (if I didn't miscalculate). But according to this there should be 1280KB flash, what is the rest used for? http://platformio.org/boards. Also why does platform.io list the devices as having 280KB RAM when this states 520KB, https://en.wikipedia.org/wiki/ESP32, does the operatingsystem use up 240KB by it self?

trycoon commented Sep 14, 2017

@me-no-dev Thanks! Looking at the partition scheme, it seems like the max program size is 81920 bytes (0x140000) since that whats fits into the ota-space. So my application is located in "ota_0" and when I update the program using OTA it will be uploaded to "ota_1" and then a pointer is changes to point to "ota_1" and the device is restarted? Next time it will upload to "ota_0" and re-point to ota_0, is that how it works?
Adding up the memory-spaces gives me 81920*2+8192+20480+4096+93952=290560 bytes (if I didn't miscalculate). But according to this there should be 1280KB flash, what is the rest used for? http://platformio.org/boards. Also why does platform.io list the devices as having 280KB RAM when this states 520KB, https://en.wikipedia.org/wiki/ESP32, does the operatingsystem use up 240KB by it self?

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Sep 14, 2017

Member

0x140000 = 1310720B (1.25MB)

that 520 is used for many things ;) not available to you

Member

me-no-dev commented Sep 14, 2017

0x140000 = 1310720B (1.25MB)

that 520 is used for many things ;) not available to you

@igrr

This comment has been minimized.

Show comment
Hide comment
@igrr

igrr Sep 14, 2017

Member

Also why does platform.io list the devices as having 280KB RAM when this states 520KB

Just to expand on what @me-no-dev said: 520KB is the total amount of RAM. That includes IRAM (instruction RAM), DRAM (data RAM), and RTC memory. "System and memory" chapter of the ESP32 Technical reference manual has a table explains this in more detail. Specifically, there is 328 KB of DRAM available on the chip (the rest is IRAM and RTC RAM). Some of that is used by the OS, some is used by tasks created during SDK startup. An empty application which just prints the amount of available heap will report ~300KB of free heap, in the latest IDF.

Enabling features such as WiFi, BT, and dual core support, reduces the amount of memory available to applications. I guess this is where the 280KB number comes from.

Member

igrr commented Sep 14, 2017

Also why does platform.io list the devices as having 280KB RAM when this states 520KB

Just to expand on what @me-no-dev said: 520KB is the total amount of RAM. That includes IRAM (instruction RAM), DRAM (data RAM), and RTC memory. "System and memory" chapter of the ESP32 Technical reference manual has a table explains this in more detail. Specifically, there is 328 KB of DRAM available on the chip (the rest is IRAM and RTC RAM). Some of that is used by the OS, some is used by tasks created during SDK startup. An empty application which just prints the amount of available heap will report ~300KB of free heap, in the latest IDF.

Enabling features such as WiFi, BT, and dual core support, reduces the amount of memory available to applications. I guess this is where the 280KB number comes from.

@trycoon

This comment has been minimized.

Show comment
Hide comment
@trycoon

trycoon Sep 14, 2017

Ok, thanks for the clarification.

trycoon commented Sep 14, 2017

Ok, thanks for the clarification.

@kendo55

This comment has been minimized.

Show comment
Hide comment
@kendo55

kendo55 Sep 30, 2017

@bbx10 Thanks, worked for me perfect, too!

kendo55 commented Sep 30, 2017

@bbx10 Thanks, worked for me perfect, too!

@lonerzzz

This comment has been minimized.

Show comment
Hide comment
@lonerzzz

lonerzzz Oct 15, 2017

Contributor

@me-no-dev @igrr What are we using as the equivalent of the file system uploader plugin for ESP32? If the update to the esp8266 file system uploader (https://github.com/esp8266/arduino-esp8266fs-plugin) is the intended solution, please add me to that project and I can make the changes.

Contributor

lonerzzz commented Oct 15, 2017

@me-no-dev @igrr What are we using as the equivalent of the file system uploader plugin for ESP32? If the update to the esp8266 file system uploader (https://github.com/esp8266/arduino-esp8266fs-plugin) is the intended solution, please add me to that project and I can make the changes.

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev
Member

me-no-dev commented Oct 16, 2017

@lonerzzz https://github.com/me-no-dev/arduino-esp32fs-plugin :) mkspiffs works since friday :)

@beegee-tokyo

This comment has been minimized.

Show comment
Hide comment
@beegee-tokyo

beegee-tokyo Oct 16, 2017

Contributor

@me-no-dev, @igrr
how can I use the plugin on PlatformIO?

Contributor

beegee-tokyo commented Oct 16, 2017

@me-no-dev, @igrr
how can I use the plugin on PlatformIO?

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Oct 16, 2017

Member

@beegee-tokyo I don't think that you can. PIO would need a custom implementation for that.

Member

me-no-dev commented Oct 16, 2017

@beegee-tokyo I don't think that you can. PIO would need a custom implementation for that.

@beegee-tokyo

This comment has been minimized.

Show comment
Hide comment
@beegee-tokyo

beegee-tokyo Oct 16, 2017

Contributor

@me-no-dev
Understand.
Found a solution to use it:

  1. Create an empty.ino file inside the src folder of the PlatformIO project.
  2. Create the data folder inside the src folder of the PlatformIO project.
  3. Move the files that should be uploaded into this data folder
  4. Start ArduinoIDE, select the empty.ino as sketch and then use Tools > ESP32 Sketch Data Upload to flash the SPIFFS image to the ESP32

;) :)

Contributor

beegee-tokyo commented Oct 16, 2017

@me-no-dev
Understand.
Found a solution to use it:

  1. Create an empty.ino file inside the src folder of the PlatformIO project.
  2. Create the data folder inside the src folder of the PlatformIO project.
  3. Move the files that should be uploaded into this data folder
  4. Start ArduinoIDE, select the empty.ino as sketch and then use Tools > ESP32 Sketch Data Upload to flash the SPIFFS image to the ESP32

;) :)

@lonerzzz

This comment has been minimized.

Show comment
Hide comment
@lonerzzz

lonerzzz Oct 16, 2017

Contributor

@me-no-dev Perfect, I am now marking SPIFFS as done in the Project tracking area. Didn't realize the UI portion was also completed.

Contributor

lonerzzz commented Oct 16, 2017

@me-no-dev Perfect, I am now marking SPIFFS as done in the Project tracking area. Didn't realize the UI portion was also completed.

@me-no-dev

This comment has been minimized.

Show comment
Hide comment
@me-no-dev

me-no-dev Oct 16, 2017

Member

@lonerzzz didn't say anything because mkspiffs was not working :) had to get that fixed first

Member

me-no-dev commented Oct 16, 2017

@lonerzzz didn't say anything because mkspiffs was not working :) had to get that fixed first

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment