LittleFS loses file content.

[HamzaHajeir]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [3.0.0]
• Development Env: [Platformio]
• Operating System: [Windows]

Settings in IDE

• Module: Nodemcu]
• Flash Mode: [dio]
• Flash Size: [4MB]
• lwip Variant: [Higher Bandwidth]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [921600] (serial upload only)

Problem Description

While using LittleFS, file is getting lost if the MCU resets.

It was early discovered by noticing some configuration are lost, caused by a bad and loosely power supply cable causes MCU to reboot several times at start.

After that, I've verified the issue, and wrote an example sketch with an external user requirement to reproduce it, which is pressing RST button quickly.

While LittleFS documentation states the following:

Power-loss resilience - littlefs is designed to handle random power failures. All file operations have strong copy-on-write guarantees and if power is lost the filesystem will fall back to the last known good state.

MCVE Sketch

An extra calls are put to produce helpful (hopefully) information to the Serial, starting with LittleFS.end();.

#include <Arduino.h>
#include <FS.h>
#include <LittleFS.h>

void listDir(const char * dirname) {
  Serial.printf("Listing directory: %s\n", dirname);

  Dir root = LittleFS.openDir(dirname);

  while (root.next()) {
    File file = root.openFile("r");
    Serial.print("  FILE: ");
    Serial.print(root.fileName());
    Serial.print("  SIZE: ");
    Serial.println(file.size());
    file.close();
  }
}


void readFile(const char * path) {
  Serial.printf("Reading file: %s\n", path);

  File file = LittleFS.open(path, "r");
  if (!file) {
    Serial.println("Failed to open file for reading");
    return;
  }

  Serial.print("Read from file: ");
  while (file.available()) {
    Serial.write(file.read());
  }
  Serial.println();
  file.close();
}

void writeFile(const char * path, const char * message) {
  Serial.printf("Writing file: %s\n", path);

  File file = LittleFS.open(path, "w");
  if (!file) {
    Serial.println("Failed to open file for writing");
    return;
  }
  if (file.print(message)) {
    Serial.println("File written");
  } else {
    Serial.println("Write failed");
  }
  delay(2000); // Make sure the CREATE and LASTWRITE times are different
  file.close();
}

void setup() {
  Serial.begin(115200);
  Serial.println("Mount LittleFS");
  if (!LittleFS.begin()) {
    Serial.println("LittleFS mount failed");
    return;
  }
  readFile("/hello.txt");
  listDir("/");
  writeFile("/hello.txt", "Hello ");
  LittleFS.end();
  if (!LittleFS.begin()) {
    Serial.println("LittleFS mount failed");
    return;
  }
  readFile("/hello.txt");
}

void loop() { }

platformio.ini:

It could contain some irrelated options,

framework = arduino
platform = espressif8266
board = nodemcuv2
monitor_speed = 115200
upload_speed = 921600
build_flags = 
	-DNDEBUG
	-D PIO_FRAMEWORK_ARDUINO_LWIP2_HIGHER_BANDWIDTH_LOW_FLASH
board_build.flash_mode = dio
board_build.filesystem = littlefs
board_build.ldscript = eagle.flash.4m1m.ld
lib_deps = 
    LittleFS
lib_ldf_mode = deep+

Debug Messages

15:14:56.092 > Reading file: /hello.txt
15:14:56.098 > Read from file: Hello 

15:14:56.100 > Listing directory: /
15:14:56.116 >   FILE: hello.txt  SIZE: 6

15:14:56.119 > Writing file: /hello.txt
15:14:56.130 > File written

15:14:58.104 >   <<<<<<CONSECUTIVE QUICK MANUAL REBOOTS VIA RST PIN>>>>>>  Mount LittleFS

15:14:58.482 > Reading file: /hello.txt
15:14:58.488 > Read from file: 

15:14:58.490 > Listing directory: /
15:14:58.506 >   FILE: hello.txt  SIZE: 0

15:14:58.509 > Writing file: /hello.txt
15:14:58.519 > File written

15:15:00.540 > Reading file: /hello.txt
15:15:00.547 > Read from file: Hello 

[earlephilhower]

This looks to be a coding issue in the MCVE. not LittleFS.

Until the File is close()ed or flush()ed, there is a small RAM buffer used to store the data sent to it. So, it looks like you've reset the system between when the print "File Written" but before the file is closed or flushed. Until either of those 2 calls are done, there are no guarantees. This is due to flash limitations and wear issues, and similar to standard PCs where there is small RAM cache that needs to be flush()ed with the proper POSIX call to guarantee it's on stable store.

[HamzaHajeir]

Thanks @earlephilhower for your information,

What I think is LittleFS wrapper library doesn't employ original littlefs correctly, somehow causing the file to be erased first before calling File::close() to commit the file.

What makes it more clear that littlefs states it falls back to last known state, which is not the case.

Power-loss resilience - littlefs is designed to handle random power failures. All file operations have strong copy-on-write guarantees and if power is lost the filesystem will fall back to the last known good state.

I'd welcome If this kind of feedback is welcomed: (removing delay() between file.print(message) and file.close().)
Or an output while some debugs are activated.

[earlephilhower]

No, that's not the case, or the way LittleFS atomicity is spec'd. LittleFS lfs_file_open() is called directly from the File::open and LFS itself does the directory update.

It can't be any other way. Say, for example, you have a 1MB file on a 1MB FS, so 0 bytes free. If open("file", "w") did not erase the existing file, you could never write to the file again. The "w" implies a 0-ing out of the dir entry, which is rewritten by LFS in an atomic way (i.e. the FS won't go inconsistent due to a partial write).

For what you want to do, the traditional thing is to have 2 files with an epoch counter in each. You then overwrite the older epoch'd file on each run. On startup you'd read both files and pick the highest epoch.

[HamzaHajeir]

The workaround could solve it, but it's not what's expected from a FS to lose an entire file.

IMO, as a library: it shouldn't be left to the user to do so.

Now it's going to be LFS related issue (not to the wrapper).