Bug: iterating a FileStore reopens the segment file for every record, costing 162 ms/record on LittleFS

[mclueppers]

Version / platform

• microStore 0.1.7 (master, 0f28567), used through microReticulum 0.5.x
• Heltec V4 (ESP32-S3, 8 MB flash), Arduino core 3.3.11 / ESP-IDF 5.5.5, pioarduino espressif32, xtensa-esp-elf gcc 14.2.0
• BasicFileStore on LittleFS (internal flash), prefix ./path_store/, segment_size=65536, segment_count=8
• 200 records in the store, reached as TypedStore<Bytes, DestinationEntry, BasicFileStore<...>>

Expected behaviour

Walking a store with begin() / ++ / operator* costs about one record read per record.

Actual behaviour

Every dereference costs a file open as well, and on LittleFS that is the expensive half. Measured on the hardware above:

	per record	64 records	200 records
observed	162 ms	10.4 s	~32 s

The calling task feeds a 30 s task watchdog once per pass, so a full walk did not finish in time and the device rebooted — every ~90 seconds, for two days, before it was traced here.

Cause

iterator::load_value() (FileStore.h:846) opens the segment file, seeks, reads and closes it, once per record:

void load_value() const {
    char name[USTORE_MAX_FILENAME_LEN];
    store_->segment_name(iv_.segment, name);

    File f = store_->_filesystem.open(name, File::ModeRead);   // line 850 — per record
    ...
    f.close();
}

operator* (line 803) calls it at every position, so a walk of N records performs N opens of the same file.

On LittleFS an open() resolves the path from the root each time, and resolving a component replays that directory's metadata log through lfs_dir_fetchmatch. That log grows with write activity until it is compacted, so a store under steady write traffic charges more and more per open while the record read itself stays the same size. The cost is therefore not constant — it degrades in service, and a freshly created store does not show it.

An ESP32 coredump taken as the watchdog fired lands in exactly that path:

#29 lfs_dir_fetchmatch
#30 lfs_dir_find
#31 lfs_file_opencfg_
#33 vfs_littlefs_open
#46 fs::FS::open (LittleFS, "./path_store/seg0.dat", "r")
#48 microStore::FileSystem::open (path="./path_store/seg0.dat", mode=ModeRead)
#49 microStore::BasicFileStore<...>::iterator::load_value ()    FileStore.h:850
#50 microStore::BasicFileStore<...>::iterator::operator* ()     FileStore.h:803
#51 microStore::TypedStore<...>::iterator::load ()              TypedStore.h:127
#52 microStore::TypedStore<...>::iterator::operator++ ()        TypedStore.h:105

Steps to reproduce

1. On an ESP32 with a BasicFileStore on LittleFS, put ~200 records and let the store take normal write traffic for a while, so the directory's metadata log is not freshly compacted.
2. Walk it and time the loop:

   for (auto it = store.begin(); it != store.end(); ++it) { auto& e = *it; (void)e; }

3. The loop takes tens of seconds. Timing a single *it shows nearly all of it inside the filesystem open(), not the read.

Impact

Any caller that walks a store on a device with a watchdog is exposed, and the exposure grows with both record count and write history. Through microReticulum this is the path table: a transport node holding a few hundred paths cannot enumerate them inside a watchdog period, and the failure appears only after the node has been in service long enough for the table to grow.

Suggested fix

Hold the segment handle for the duration of the iteration instead of reopening per record, swapping it when a walk crosses a segment boundary. On the hardware above that took a record from 162 ms to 32 ms, and the 200-record walk from ~32 s to ~6.4 s. No API change.