Add LittleFS port as a component (IDFGH-3514)

[BrianPugh]

Added my port of LittleFS for esp-idf. LittleFS significantly outperforms SPIFFS in most common tasks, and is much more robust when compared to FAT.

Benefits of my port:

• Structured nearly identically to the SPIFFS port, so anyone that was comfortable in that repo should feel comfortable in this repo
• Pretty comprehensive unit tests
• Some systematic improvements/efficiencies such as name-hashing, no-file-limit, nonce-metadata, etc.
• Proper use of submoduling

Some potential areas for improvement:

• Documentation (However the API is the same as SPIFFS with a few exceptions noted in the README.md)
• For the partition_label required to be specified (instead of allowing it to be NULL) in the configuration struct, this could be mitigated by adding a littlefs subdatatype to the partition table. Other ports of LittleFS have done this.

[lorol]

@BrianPugh
It is better to keep the partition scheme fully over SPIFFS. Less-likely one will want to have both FS on same project. Also an Arduino SPIFFS is "THE" partition for data folder and tools work well if they move just LittleFS inside SPIFFS. This approach is already used in ESP8266. Why to split?

[BrianPugh]

I'm thinking that my code will still allow the use of a SPIFFS partition if specified by name, this change would be just to support NULL (defaults to first found LITTLEFS partition) within esp-idf projects to more perfectly mirror the SPIFFS api.

[Alvin1Zhang]

Thanks for your contribution.

[igrr]

@BrianPugh What is your opinion on keeping this feature as a standalone component? We will soon announce a component manager for IDF, and will start moving many of IDF components out of the IDF repository. I think LittleFS is a good candidate

The reason we can't accept the PR as is, is that we don't have the control over https://github.com/joltwallet/esp_littlefs.git repository. If we get an issue report for the LittleFS feature, we would depend on the owner of the repository to merge a fix. To support multiple IDF branches (release/v4.2, release/v4.3, future releases), we would also need to maintain corresponding branches in the submodule. Plus, as you correctly point out in the linked arduino-esp32 PR, we would also require an API reference page and some examples to be added for such feature — but this is something we could help with, provided that the other question is resolved.

Another question, have you looked into whether it is possible to use LittleFS with flash encryption?

[BrianPugh]

@igrr

I'm personally fine with it being a standalone component, but in the linked arduino discussion, it seems like it would clean a lot of duplicate code, and allow stuff to be properly submoduled so things like bug-fixes would propagate a lot faster/easier.

As for control on the repo, we can of course always have Espressif fork my repo.

Making sure LittleFS works with flash encryption is on my todo list. Currently I left in the same encryption assertions as SPIFFS just to be safe. I'll try and mess around with it this weekend. I haven't actually dived deep into flash encryption yet; whats the reason that SPIFFS cannot use flash encryption?

[atanisoft]

@BrianPugh The reason for the PR here was mostly for the same reasons as @igrr is stating that it can't be accepted here (unfortunately). The current PR against arduino-esp32 duplicates your repository into arduino-esp32 and that will cause problems as the sources diverge over time. In the case of SPIFFS arduino-esp32 is leveraging the IDF SPIFFS component so as not to duplicate the code, similarly with most other FS types.

[igrr]

whats the reason that SPIFFS cannot use flash encryption?

SPIFFS relies on the "NAND write behavior" of NOR flash, which is that you can selectively program individual bits of flash from 1 to 0. This is not possible when flash encryption is enabled, because flash encryption works in 128-bit blocks. It is not possible to change a single bit from 1 to 0 in the plaintext, because this will result in a pseudo-random change of the ciphertext, with some bits changing from 0 to 1. The latter is not possible in normal NOR flash without an erase operation. But setting individual bits is used exactly for the purpose of avoiding additional erase operations, and is also an important part of power failure resistance and wear leveling.

We will soon add a new API to esp_partition, which will allow doing non-encrypted writes to an encrypted partition. However this can only help if the data is structured such that "raw" writes and "encrypted" writes never land into the same 16-byte block.

The current PR against arduino-esp32 duplicates your repository into arduino-esp32 and that will cause problems as the sources diverge over time.

I think for Arduino, the best option would be to register the original repository (https://github.com/joltwallet/esp_littlefs.git) as an Arduino library. I understand that @BrianPugh has no interest in maintaining Arduino support. In this case, another (wrapper) repository could be created, where esp_littlefs would be imported as a git submodule or subtree, in addition to Arduino-specific source/header/metadata files. I am not sure whether Arduino Library Manager supports repositories which contain submodules, thus mentioning subtree as a backup option. Both submodules and subtrees preserve the relation to the original repository, making it easy to update the library.

[atanisoft]

I am not sure whether Arduino Library Manager supports repositories which contain submodules, thus mentioning subtree as a backup option.

Arduino Library Manager does not support sub-modules or trees. It is a flat repository setup only.

[igrr]

Regarding subtrees — that's odd, from Git perspective a subtree is indistinguishable from regular files committed into the repository. Does this not work because the source directories need to be just one level deep, or something similar?

[atanisoft]

Does this not work because the source directories need to be just one level deep, or something similar?

Their requirements to list in the library manager are a bit odd to say the least. They won't list any library that includes a submodule, subtree, or anything that you won't get via a zip file download from github (which excludes all fancy bits that rely on .submodules or similar marker files that the git cli would reference)

Is the long-term plan for IDF to not have many of the current components in the core deliverable? Is there a list of components that are going to externalized?

[igrr]

They won't list any library that includes a submodule, subtree, or anything that you won't get via a zip file download from github

The content of a subtree or a subrepo is included into a plain .zip download on Github — that I am pretty sure about, but perhaps there is an additional limitation on github Arduino library manager side. I'll try this out if i get some time...

Is the long-term plan for IDF to not have many of the current components in the core deliverable? Is there a list of components that are going to externalized?

There is no list of components which will be made standalone, yet. I can name a few components which are more likely to be moved out than the rest: expat, coap, cbor, freemodbus, libsodium, jsmn, mqtt, nghttp, plus the components maintained under the examples directory.

[atanisoft]

There is no list of components which will be made standalone, yet. I can name a few components which are more likely to be moved out than the rest: expat, coap, cbor, freemodbus, libsodium, jsmn, mqtt, nghttp, plus the components maintained under the examples directory.

ok these make sense, what about the various VFS adapters (SPIFFS, SD, FFat, etc)?

[BrianPugh]

@igrr So i haven't really dived deep into the under workings of littlefs, but I just tested using flash encryption in DEVELOPMENT mode and everything worked. However, as part of my unit-tests I also have benchmark tests that compare performance against FAT and SPIFFS. In this configuration, SPIFFS still ran, which is a bit concerning.

Basically, what further actions should i take to make sure this works with encryption?

[igrr]

Just to check, have you marked the filesystem partition "encrypted" in the partition table, like it is done here?

esp-idf/examples/security/flash_encryption/partitions_example.csv

Line 4 in 625bd5e

storage, data, 0xff, 0xf000, 0x1000, encrypted

[BrianPugh]

@igrr I forgot to do that, so that would explain that.

Related: it seems like I may have bricked my device (it's boot-looping). This is no big deal because I have a ton of spares, but just trying to figure out if I actually bricked it and how to prevent it in the future. I know flash encryption and stuff burns efuses, but I thought DEVELOPMENT mode was safe for reflashes and all that. Do I have any misunderstanding here?

[igrr]

Could you please run espefuse.py summary and post the output? This may help figure out both what has happened, and whether the device is recoverable.

[BrianPugh]

Here's the output of espefuse.py summary

Detecting chip type... ESP32
espefuse.py v3.0-dev
EFUSE_NAME (Block)                       Description  = [Meaningful Value] [Readable/Writeable] (Hex Value)
----------------------------------------------------------------------------------------
Security fuses:
FLASH_CRYPT_CNT (BLOCK0):                Flash encryption mode counter                      = 1 R/W (0b0000001)
UART_DOWNLOAD_DIS (BLOCK0):              Disable UART download mode (ESP32 rev3 only)       = False R/W (0b0)
FLASH_CRYPT_CONFIG (BLOCK0):             Flash encryption config (key tweak bits)           = 15 R/W (0xf)
CONSOLE_DEBUG_DISABLE (BLOCK0):          Disable ROM BASIC interpreter fallback             = True R/W (0b1)
ABS_DONE_0 (BLOCK0):                     secure boot enabled for bootloader                 = False R/W (0b0)
ABS_DONE_1 (BLOCK0):                     secure boot abstract 1 locked                      = False R/W (0b0)
JTAG_DISABLE (BLOCK0):                   Disable JTAG                                       = True R/W (0b1)
DISABLE_DL_ENCRYPT (BLOCK0):             Disable flash encryption in UART bootloader        = False R/W (0b0)
DISABLE_DL_DECRYPT (BLOCK0):             Disable flash decryption in UART bootloader        = True R/W (0b1)
DISABLE_DL_CACHE (BLOCK0):               Disable flash cache in UART bootloader             = True R/W (0b1)
BLK1 (BLOCK1):                           Flash encryption key
   = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? -/-
BLK2 (BLOCK2):                           Secure boot key
   = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W
BLK3 (BLOCK3):                           Variable Block 3
   = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W

Efuse fuses:
WR_DIS (BLOCK0):                         Efuse write disable mask                           = 128 R/W (0x0080)
RD_DIS (BLOCK0):                         Efuse read disable mask                            = 1 R/W (0x1)
CODING_SCHEME (BLOCK0):                  Efuse variable block length scheme
   = NONE (BLK1-3 len=256 bits) R/W (0b00)
KEY_STATUS (BLOCK0):                     Usage of efuse block 3 (reserved)                  = False R/W (0b0)

Config fuses:
XPD_SDIO_FORCE (BLOCK0):                 Ignore MTDI pin (GPIO12) for VDD_SDIO on reset     = False R/W (0b0)
XPD_SDIO_REG (BLOCK0):                   If XPD_SDIO_FORCE, enable VDD_SDIO reg on reset    = False R/W (0b0)
XPD_SDIO_TIEH (BLOCK0):                  If XPD_SDIO_FORCE & XPD_SDIO_REG                   = 1.8V R/W (0b0)
CLK8M_FREQ (BLOCK0):                     8MHz clock freq override                           = 51 R/W (0x33)
SPI_PAD_CONFIG_CLK (BLOCK0):             Override SD_CLK pad (GPIO6/SPICLK)                 = 0 R/W (0b00000)
SPI_PAD_CONFIG_Q (BLOCK0):               Override SD_DATA_0 pad (GPIO7/SPIQ)                = 0 R/W (0b00000)
SPI_PAD_CONFIG_D (BLOCK0):               Override SD_DATA_1 pad (GPIO8/SPID)                = 0 R/W (0b00000)
SPI_PAD_CONFIG_HD (BLOCK0):              Override SD_DATA_2 pad (GPIO9/SPIHD)               = 0 R/W (0b00000)
SPI_PAD_CONFIG_CS0 (BLOCK0):             Override SD_CMD pad (GPIO11/SPICS0)                = 0 R/W (0b00000)
DISABLE_SDIO_HOST (BLOCK0):              Disable SDIO host                                  = False R/W (0b0)

Identity fuses:
MAC (BLOCK0):                            Factory MAC Address
   = b4:e6:2d:bf:68:4d (CRC 0x04 OK) R/W
MAC_CRC (BLOCK0):                        CRC8 for factory MAC address                       = 4 R/W (0x04)
CHIP_VER_REV1 (BLOCK0):                  Silicon Revision 1                                 = True R/W (0b1)
CHIP_VER_REV2 (BLOCK0):                  Silicon Revision 2                                 = False R/W (0b0)
CHIP_VERSION (BLOCK0):                   Reserved for future chip versions                  = 2 R/W (0b10)
CHIP_PACKAGE (BLOCK0):                   Chip package identifier                            = 0 R/W (0b000)
MAC_VERSION (BLOCK3):                    Version of the MAC field                           = 0 R/W (0x00)

Calibration fuses:
BLK3_PART_RESERVE (BLOCK0):              BLOCK3 partially served for ADC calibration data   = False R/W (0b0)
ADC_VREF (BLOCK0):                       Voltage reference calibration                      = 1086 R/W (0b10010)

Flash voltage (VDD_SDIO) determined by GPIO12 on reset (High for 1.8V, Low/NC for 3.3V).

And, even though it's probably of no help, here's the output of my device bootlooping:

Hard resetting via RTS pin...
MONITOR
--- idf_monitor on /dev/ttyUSB0 230400 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
f~f~x~x怘怘~ff~fff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~f~f~x~x怘怘~fff`ffff`ff`~fxff`ff`f`fxfxxf~ffx~

[igrr]

espefuse.py output looks fine — it is correct for the "Development" mode of flash encryption.
I see you are running idf_monitor with 230400 baud rate. This is the likely cause of garbage output, as the ROM bootloader prints messages at 115200 baud. If you change the monitor baud rate to 115200, i think you should be able to see readable output.
After that, try running idf.py encrypted-flash monitor. This should flash&encrypt the bootloader, partition table, and the app, and open the monitor.

[BrianPugh]

thanks! I'll mess around with this more later today. Hopefully this unblocks me, I'll test to see if my littlefs port still works with encryption, and I'll comment back on the results.

[BrianPugh]

@igrr So setting the baudrate to 115200 helped solve my boot-loader logging issue. It was reporting back flash read err, 1000. This makes sense since i was just classic flashing my app when I really had to use encrypted-flash as you mentioned. Also, not setting the encrypted flag in the partition table was definitely an issue. Previously without the encrypted partition flag, everything ran fine, but then with the flag my code aborted due to the encryption check (which is great!).

I removed the portion of code that aborted if encryption is enabled, and all my unit-tests (which are mostly similar to the SPIFFS unit-tests) passed, so I think everything is fine. I've committed this to the master branch of the repo. Can you think of anything else I should test/look out for? I'm optimistic since my unit-tests work and the author of littlefs says that they make as little assumptions about the underlying storage as possible.

[igrr]

@BrianPugh That's great news! Indeed it seems that setting the read and write sizes to be multiple of 32 bytes will make LittleFS compatible with the hardware flash encryption.

I think the remaining questions to work out are:

1. Decide where this component will be maintained: as an Espressif fork of your project, as source files in IDF, or some other way. Where will the development happen, what will be the strategy for merging upstream changes, and so on.
2. Review esp_littlefs.c. I only had a cursory look, but would like to do a full review, as some parts of the code are not trivial. Sadly it is not possible to do in this PR, as the component is added as a submodule.
3. Review Kconfig options and their default values.
4. Find a solution for building LittleFS filesystems on the host. There is an mklfs tool in Lua-RTOS project, but the downside is that it is written in C and needs to be built with the same set of options as littlefs component. Since IDF doesn't require a host compiler, this is problematic. For SPIFFS partition generation we have developed a Python tool, spiffsgen.py. This is not a blocker for merging the PR, as the tool could be added later. We just need to see how feasible it would be to write something similar for LittleFS.
5. If p.4 is resolved, add a CMake helper function to automate generation of LittleFS partitions, similar to

   esp-idf/components/spiffs/project_include.cmake

   Line 5 in 625bd5e

   function(spiffs_create_partition_image partition base_dir)

6. Add an API reference page to the documentation. We can handle this part when merging your PR.
7. Add an example project, probably similar to the existing SPIFFS example.
8. Update other existing IDF examples to use LittleFS instead of SPIFFS, where it makes sense.

[lorol]

@BrianPugh @igrr
Entirely up to you Brian.
Below are references of what I touched last weeks.
My modest knowledge' summary:

• It works as a hack directly dropped as I showed, but needs paths adjustment (and adding defines)
• I tried down to Arduino core 1.0.4 release (IDF 3.2) - it builds just needs the utime removal
• see the mklittlefs in links below, it is compatible
• the Arduino FS uploader (java) also was too easy to adapt
• tried with PIO (arduino) - works as well mkliitlefs needs to be added
• name LITTLEFS or LittleFS, maybe the second one is more correct?
• It looks like the easiest is to overlap SPIFFS and leave as it is for partition scheme naming and "data" folder manipulation (only one, either LittleFS or SPIFFS in a project)
• I have no idea about Licenses compatibility and I have very limited time, skills and resources but if you lead it I can help with some stuff of course,
• I don't care if it would be a separate Arduino library or through IDF component then in Arduino core as long as it gets out and people enjoy it :)

https://github.com/lorol/LITTLEFS
https://github.com/lorol/arduino-esp32littlefs-plugin or :) even me-no-dev/arduino-esp32fs-plugin#23

espressif/arduino-esp32#3765
espressif/arduino-esp32#4096
#5469
https://github.com/earlephilhower/mklittlefs
me-no-dev/ESPAsyncWebServer#792

[BrianPugh]

@igrr Addressing your numbered comments:

1. I can give you guys access to my repo. Alternatively, an espressif-fork would work as well so you guys have complete control.
2. Would certainly require a lengthy review, but i'd say 90% of it is one-to-one with SPIFFS. If it makes the review process easier, I'd recommend starting at commit acacb178. After that commit @X-Ryl669 added some features that improved performance, which is not similar to the SPIFFS-codebase.
3. Makes sense.
4. See information @lorol provided. Also a quick googling found littlefs-python which should accomplish your goals in just a few lines of code.
5. Yup!
6. Sounds Good
7. Should be little more than find/replacing spiffs with littlefs
8. 👍

[X-Ryl669]

I can confirm that one of the selling point of LittleFS compared to SPIFFS is that LittleFS does not make any assumption on the block device (just that it's a flash than need erasing before any write) see here.

IMHO, littlefs is way better than SPIFFS in many points and I think it's bad to try to fit LittleFS into the limitations of SPIFFS.
Flash encryption is one example, but you also have access to metadata, etc.. that does not exist in SPIFFS code. Similarly, having a format function in the embedded code is much better than trying to replicate the requirement of SPIFFS's python tool.

It also exist a FUSE wrapper for littlefs so you can manipulate your ESP's filesystem like a standard filesystem (without having to create a limited python tool). And for those without FUSE (like windows user), you can use a javascript wrapper directly in your browser (but it's more low level) so you don't even need a compiler on your machine.

Also, the current virtual filesystem in ESP is limited to 64 files descriptors (IIRC) and leaks locks on those since it's calling lock_acquire on uninitialized locks (which in turns call lock_init that's allocating a lock via FreeRTOS) but never deallocate them. I'm not sure it's intentionnal or simply an mistake but it will likely break for heavy file jobs.

[tore-espressif]

Hello @BrianPugh

It's been a while, since you posted this, but if you are still interested I'd love to help you to get your LittleFS to our new Espressif's idf-component-manager service,

Idf-component-manager is a new function that allows ESP-IDF users to integrate software package into their project seamlessly. LittleFS could be one of the first external components hosted there.

More information about idf-component manager can be found in Espressif API guide or PyPi registry. The component service itself is hosted here.

Let me know if you agree and I can push a PR to LittleFS repository, so the sw package is deployed after you push to master branch. Or you can upload it manually whenever you see it fit.

[BrianPugh]

@tore-espressif sure! A PR would be lovely; in the PR just let me know what actions need to be taken on my end.