Limit SPIFFS use (ESP32)

[DE-cr]

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/spiffs.html

"SPIFFS is able to reliably utilize only around 75% of assigned partition space."

[DE-cr]

Although the official sources say that only 75 % of SPIFFS can be reliably used, I'll keep the old #define MINIMUM_FREE_SPACE_ON_SD 10000 instead of 1.5 MB * 0,25 == 393216 for my system.

Reason: even 1.5 MB isn't plenty for data logging (when you don't have an SD card in your system), and having that reduced to <1.2 MB for "reliable use" only makes matters worse in that regard. Also, in my "testing" (extremely limited, just one test case!), using more than 75 % of SPIFFS did work:

With 631516 bytes of SPIFFS space available, I tried to use /DK1 (c.f. #542 (comment)) on a datalog.txt of 740704 bytes, only to receive the error message "Not enough space on device" with the 25 % reserve requirement.

After reducing the reserve to the original 10000 bytes (< 1.6 %), /DK1 did work - albeit it took two attempts, and it did take longer than expected:

$ time wget -qO- bsb-lan/DK1
.......... .......... ...
Error writing datalog

real	0m21,897s
user	0m0,001s
sys	0m0,009s
$ time wget -qO- bsb-lan/DK1
.......... .......... .....
Success

real	0m42,337s
user	0m0,001s
sys	0m0,008s

For completeness:
After /DK1, my datalog.txt was 263379 bytes, and free SPIFFS was 1111679.
With a 25 % reserve, "not enough space" was correct: (datalog:263379 + reserve:393216 == 656595) > spiffs:631516.

It's up to you to decide whether to merge this PR or not.

[fredlcore]

The question is whether a switch to LittleFS would make more sense?

[DE-cr]

Having just read about LittleFS, I'd say: In the long term, using it seems to be the way to go. Right now, it seems easier to use SPIFFS instead, and SPIFFS seems to be "good enough" in most cases.

[fredlcore]

I haven't seen the need for it so far as well, but if it means that the already limited space on a ESP32 without SD card is further reduced by 25%, it might be worth looking at it more closely. I read somewhere that the library calls are mostly compatible, so it might be as easy as changing the included library, but I'll see when I find time to look at it more closely...

[fredlcore]

It actually is just a drop-in replacement. Tested it on my ESP32 Olimex with its internal memory and logging, displaying and removing log files work fine. So I'll keep the minimum at 10000 bytes for now.

[fredlcore]

Since you did a few tests regarding speed, I wonder if LittleFS will produce noticeable gains as well...

[DE-cr]

First simple test to compare LittleFS to SPIFFS, in a sketch doing just that:

...and yes, it's really as simple as #define SPIFFS LittleFS and #including a different header file - once you've added the esp_littlefs library in your Arduino IDE.
One more timing comparison: formatting took 5.786 s with LittleFS vs. 26.536 s with SPIFFS.

[fredlcore]

Shouldn't be necessary to add LittleFS as it is part of the ESP core v2. If that's the library you've added, that's the predecessor and it's outdated:
https://github.com/lorol/LITTLEFS

[DE-cr]

OK, I've just searched for "littlefs" in the library manager of my Arduino 1.8.19 IDE: nothing found.
Maybe I didn't install anything extra?

[DE-cr]

Using a different computer, I can confirm that I didn't have to install LittleFS. :)

Some more timings:

• For a 512 KB file, and doing I/O in 4 KB chunks, first writing that file (right after creating the file system) takes about 3.3 s using SPIFFS vs. 1.8 s using LittleFS.
• Copying that file at first takes about 5.7 s / 2.1 s.
• After just a few repetitions, write/copy times reach a constant 20.5 s for SPIFFS / 9 s for LittleFS.
• Reading that file takes a constant 330 ms / 210 ms.
• Eventually removing the file takes 2 s using SPIFFS vs. 7 ms (!) with LittleFS.

=> In my tests, LittleFS has always been considerably faster than SPIFFS.

Also, with LittleFS files up to 1400 KB could be written, as opposed to just 1200 KB with SPIFFS. => #557 (comment)

Considering that LittleFS doesn't require an extra installation and that the code change would be trivial, you should definitely consider changing bsb-lan usage of SPIFFS to LittleFS.

I have not done any reliability testing for LittleFS, though!
However, from what I can see on the net, it seems to be trustworthy.

Do you want me to prepare a pull request? :)

[fredlcore]

I've already included in the master repo when I closed this PR ;)...

[fredlcore]

But thanks for testing, the advantages are more than obvious now...