Skip to content

fix(stm32wl): recover from littlefs internal corruption instead of hanging - #11230

Open
ndoo wants to merge 2 commits into
meshtastic:developfrom
meshmy:fix/stm32wl-littlefs-no-assert
Open

fix(stm32wl): recover from littlefs internal corruption instead of hanging#11230
ndoo wants to merge 2 commits into
meshtastic:developfrom
meshmy:fix/stm32wl-littlefs-no-assert

Conversation

@ndoo

@ndoo ndoo commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Problem

LFS_ASSERT (src/platform/stm32wl/littlefs/lfs_util.h) was a plain assert(), which on STM32WL hangs forever with no diagnostic — __wrap___assert_func (src/platform/stm32wl/main-stm32wl.cpp) is while(true);, no reset, nothing printed.

STM32_LittleFS::begin() is already designed to treat filesystem corruption as recoverable — format and retry, see fsFormat()/NodeDB::saveToDisk() — but that only works if lfs_mount() cleanly returns an error code. An internal littlefs consistency check failing (metadata pair/CRC/block-allocator invariants) never returns at all; it falls straight into the assert hang. A bad flash sector or power loss mid-write could permanently brick a device that would otherwise have recovered via the existing reformat path.

nRF52 already hit this exact problem and fixed it: LFS_NO_ASSERT + a custom lfs_assert() that reboots into a reformat (#3818). STM32WL had no equivalent — it was fully exposed to the bug nRF52 already solved.

Fix

Port the nRF52 approach to STM32WL:

  • variants/stm32/stm32.ini: add -DLFS_NO_ASSERT.
  • src/platform/stm32wl/littlefs/lfs_util.h: route LFS_ASSERT through a custom lfs_assert() hook instead of disabling the check outright (matching the semantics of nRF52's Adafruit-derived header, not just "turn assertions off").
  • src/platform/stm32wl/main-stm32wl.cpp: lfs_assert() logs the corruption and requests a reformat on the next boot via a .noinit SRAM magic value — the same mechanism already used in this file for the DFU bootloader redirect, chosen specifically because backup/TAMP registers don't reliably survive a soft reset in this toolchain (see the existing comment above g_bootloaderMagic) — then reboots. preFSBegin() (the existing weak hook fsInit() already calls before mounting) sees the magic value, reformats via the existing fsFormat() path, and clears it. A plain (non-.noinit) retry-delay static throttles repeated reformat/reboot cycles within one power-on session, mirroring nRF52's lfs_assert().

Unlike nRF52 (a third-party Adafruit library, patched via a -include override specifically to avoid forking it), STM32WL's littlefs copy is already a project-owned vendored file, so lfs_util.h is edited directly — no override hack needed.

Test plan

  • pio run -e wio-e5 / pio run -e rak3172 — build clean.
  • Hardware (wio-e5, ST-Link/SWD): flashed the committed fix, confirmed normal boot/operation (meshtastic --info healthy, radio init succeeds).
  • Hardware (wio-e5), real on-disk corruption, not a synthetic trigger: halted the MCU via SWD, read the live LittleFS partition, and used a small script (reimplementing the vendored lfs_crc()/directory-commit format) to corrupt a real file's CTZ head pointer (channels.proto's head field, 8 → 9999 — a block number past block_count=56) while recomputing a matching CRC-32 so the corrupted commit still passes littlefs's own validation. Wrote it back and reset with no gap. Confirmed over serial the actual internal littlefs assertion fired with its genuine condition text:
    Load /prefs/channels.proto
    LittleFS corruption detected: block < lfs->cfg->block_count
    
    followed by reboot (not a hang) → "NOTE! Record critical error 13""LittleFS format complete; restoring default settings" → filesystem came back empty and every file correctly fell back to Install default ... → normal boot continued, radio init succeeded. This is the exact bug being fixed, reproduced from real corrupted on-disk data rather than a manual function call, and the exact recovery path this PR adds.
  • First attempt at real corruption (corrupting a directory entry's pointer instead of a file's) was silently healed by littlefs's own multi-commit save sequence before it was ever read back — informative in its own right (shows the depth of the existing CRC-validated commit log's defense in depth), but not a test of this PR. Retargeting a file's CTZ head (read on every boot, before any new commits) avoided that race.

🤝 Attestations

  • I have tested that my proposed changes behave as described.
  • I have tested that my proposed changes do not cause any obvious regressions on the following devices:
    • Heltec (Lora32) V3
    • LilyGo T-Deck
    • LilyGo T-Beam
    • RAK WisBlock 4631
    • Seeed Studio T-1000E tracker card
    • Other (please specify below): wio-e5 — build + hardware verified per test plan above. rak3172 build-verified only.

Summary by CodeRabbit

  • Bug Fixes
    • Improved recovery from LittleFS storage corruption on STM32WL devices.
    • Prevented devices from hanging during filesystem integrity check failures.
    • Added automatic, next-boot reformat and reboot when corruption is detected, restoring defaults.
    • Added safeguards to throttle repeated recovery attempts and reduce flash wear.
  • New Features
    • Enabled recoverable filesystem assertions on STM32 builds, with failure details handled safely instead of stalling.

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

STM32WL LittleFS assertions now invoke a recovery handler that records corruption, reboots the device, and formats the filesystem during the next boot. The STM32 build enables this behavior through LFS_NO_ASSERT.

Changes

STM32WL LittleFS recovery

Layer / File(s) Summary
Route LittleFS assertions to recovery
src/platform/stm32wl/littlefs/lfs_util.h, variants/stm32/stm32.ini
STM32 builds enable LFS_NO_ASSERT, and failed LFS_ASSERT expressions are passed to lfs_assert.
Recover corrupted filesystem on reboot
src/platform/stm32wl/main-stm32wl.cpp
The assertion handler records corruption, throttles reset attempts, and formats the filesystem during the next boot.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LittleFS
  participant lfs_assert
  participant SystemReset
  participant preFSBegin
  participant fsFormat
  LittleFS->>lfs_assert: failed expression text
  lfs_assert->>SystemReset: set corruption flag and request reset
  SystemReset->>preFSBegin: next boot
  preFSBegin->>fsFormat: format corrupted filesystem
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly matches the main change: STM32WL LittleFS now recovers from corruption instead of hanging.
Description check ✅ Passed The description is structured and complete, covering the problem, fix, test plan, and attestations with device coverage notes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

⚡ Try this PR in the Web Flasher

Flash this PR in the Web Flasher

firmware commit boards expires

Warning

This is an automated, unreviewed CI test build. Back up your device configuration
before flashing, and only flash devices you are able to recover.

Supported boards built by this PR (31)
Device Board Platform
Crowpanel Adv 3.5 TFT elecrow-adv-35-tft esp32-s3
Heltec HT62 heltec-ht62-esp32c3-sx1262 esp32-c3
Heltec Mesh Node 096 heltec-mesh-node-t096 nrf52840
Heltec Mesh Node T1 heltec-mesh-node-t1 nrf52840
Heltec Mesh Node T114 heltec-mesh-node-t114 nrf52840
Heltec V3 heltec-v3 esp32-s3
Heltec V4 heltec-v4 esp32-s3
Meshnology W10 meshnology_w10 esp32-s3
Meshnology W12 meshnology_w12 esp32-s3
Raspberry Pi Pico pico rp2040
Raspberry Pi Pico W picow rp2040
RAK WisMesh Pocket V3 rak_wismesh_pocket nrf52840
RAK WisMesh Pod rak_wismesh_pod nrf52840
RAK WisMesh Repeater Mini V2 rak_wismesh_repeater_mini nrf52840
RAK WisMesh Tag rak_wismeshtag nrf52840
RAK WisBlock 11200 rak11200 esp32
RAK WisBlock 11310 rak11310 rp2040
RAK3312 rak3312 esp32-s3
RAK WisBlock 4631 rak4631 nrf52840
Seeed SenseCAP Mesh-Tracker-X1 seeed_mesh_tracker_X1 nrf52840
Seeed Wio Tracker L1 seeed_wio_tracker_L1 nrf52840
Seeed Xiao NRF52840 Kit seeed_xiao_nrf52840_kit nrf52840
Seeed Xiao ESP32-S3 seeed-xiao-s3 esp32-s3
Station G2 station-g2 esp32-s3
Station G3 station-g3 esp32-s3
LILYGO T-Deck t-deck-tft esp32-s3
LILYGO T-Echo t-echo nrf52840
LILYGO T-Echo Plus t-echo-plus nrf52840
LILYGO T-Impulse Plus t-impulse-plus nrf52840
LilyGo T3-C6 tlora-c6 esp32-c6
Seeed SenseCAP T1000-E tracker-t1000-e nrf52840

Build artifacts expire on 2026-08-26. Updated for 35254ca.

@ndoo
ndoo force-pushed the fix/stm32wl-littlefs-no-assert branch from 24f2137 to 6f72258 Compare July 26, 2026 14:57
@ndoo

ndoo commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/platform/stm32wl/littlefs/lfs_util.h`:
- Around line 77-79: Update the LFS_ASSERT macro in lfs_util.h to use a
do-while(0) wrapper around its conditional assertion call, ensuring it expands
as a single statement and cannot capture a caller’s else branch.

In `@src/platform/stm32wl/main-stm32wl.cpp`:
- Around line 191-200: Replace the raw msUntilFormattingAgain and millis()
backoff logic in lfs_assert with the repository Throttle helper. Configure and
use Throttle to enforce the reformat delay safely across millis() rollover,
preserving the existing warning and delay behavior when formatting is throttled.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: fe6a594a-6f9f-47d7-8b49-ab25df9420c1

📥 Commits

Reviewing files that changed from the base of the PR and between 4800484 and 6f72258.

📒 Files selected for processing (3)
  • src/platform/stm32wl/littlefs/lfs_util.h
  • src/platform/stm32wl/main-stm32wl.cpp
  • variants/stm32/stm32.ini

Comment thread src/platform/stm32wl/littlefs/lfs_util.h Outdated
Comment thread src/platform/stm32wl/main-stm32wl.cpp
…nging

LFS_ASSERT (src/platform/stm32wl/littlefs/lfs_util.h) was a plain assert(),
which on STM32WL hangs forever with no diagnostic (__wrap___assert_func is
while(true);, see main-stm32wl.cpp). STM32_LittleFS::begin() is already
designed to treat corruption as recoverable - format and retry, see
fsFormat()/NodeDB::saveToDisk() - but that only works if lfs_mount() cleanly
returns an error. An internal littlefs consistency check failing (metadata
pair/CRC/block-allocator invariants) never returns at all, so a bad flash
sector or power loss mid-write could permanently brick a device that would
otherwise have recovered via the existing reformat path.

nRF52 already hit this and fixed it (LFS_NO_ASSERT + a custom lfs_assert()
that reboots into a reformat, see meshtastic#3818). Port the same
approach to STM32WL: LFS_NO_ASSERT routes LFS_ASSERT through a custom
lfs_assert() instead of disabling the check outright, and lfs_assert()
requests a reformat-on-next-boot via a .noinit SRAM magic value (the same
mechanism already used for the DFU bootloader redirect in this file, chosen
specifically because backup/TAMP registers don't reliably survive a soft
reset in this toolchain) and reboots, rather than trying to reformat
littlefs from inside its own possibly-mid-operation callback.

Unlike nRF52 (a third-party Adafruit library patched via a -include
override so as not to fork it), STM32WL's littlefs copy is already a
project-owned vendored file, so lfs_util.h is edited directly.

Assisted-by: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Andrew Yong <me@ndoo.sg>
@ndoo
ndoo force-pushed the fix/stm32wl-littlefs-no-assert branch from 6f72258 to 230fcda Compare July 26, 2026 15:16
@ndoo

ndoo commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both review comments: wrapped `LFS_ASSERT` in `do { ... } while (0)` so it's statement-safe, and replaced the raw `millis()` deadline arithmetic in `lfs_assert()`'s reformat backoff with `Throttle::isWithinTimespanMs()` (rollover-safe). Rebuilt and reflashed to wio-e5 to confirm normal boot after the change.

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

@ndoo Thanks—both fixes and the Wio-E5 verification are noted. I’ll review the updated changes.


If you found this review helpful, would you consider giving us a shout-out on X?

Thank you for using CodeRabbit!

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@ndoo
ndoo marked this pull request as ready for review July 26, 2026 15:33
@ndoo

ndoo commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Hardware test: real on-disk corruption, not a synthetic trigger

DUT setup

  • Board: Seeed Studio wio-e5 dev board (STM32WLE5CC, 256 KB flash)
  • Programmer: ST-Link V2 (SWD only — the wio-e5's onboard USB-serial is not wired to the STM32 UART bootloader, so SWD is the only way to flash/debug this board)
  • Console: wio-e5's onboard CP2102N USB-serial, used for the meshtastic API/log connection independent of the SWD debug link
  • Firmware: this PR's commit, built with DEBUG_MUTE temporarily undefined and PIO_FRAMEWORK_ARDUINO_NANOLIB_FLOAT_PRINTF enabled (a handful of unrelated features excluded to keep it under the flash budget with logs compiled in) — a test-only build config, not part of the diff

Method

The goal was to trigger the actual LFS_ASSERT this PR fixes via genuinely corrupted on-disk data, not by calling lfs_assert() directly. littlefs v1's on-disk format is a CRC-32-validated commit log, so a naive random-byte overwrite just gets rejected as invalid at the next commit read (already-existing, already-robust behavior, unrelated to this PR — see below). To reach the specific bug this PR fixes, the corruption has to be CRC-valid — i.e. it has to look like a legitimate commit to littlefs while still encoding a bad pointer.

Steps:

  1. Reformatted the device to a known state, then halted the MCU over SWD immediately after boot (before any further background saves could touch the filesystem).
  2. Read the live LittleFS partition (14 KB, last 7 flash pages) and located channels.proto's directory entry — a REG entry with a head field (the block number where the file's data starts) of 8.
  3. Wrote a small script that reimplements the vendored lfs_crc() (the exact nibble-table CRC-32 in lfs_util.c) and the directory-commit layout (rev/size/tail[2] header + entries + trailing CRC, per lfs_dir_commit()/lfs_dir_fetch() in lfs.c). Used it to change head from 8 to 9999 (past block_count=56) and recompute a matching trailing CRC-32, so the corrupted commit still validates.
  4. Erased and reprogrammed just that one flash page with the corrupted content while the core was still halted, then reset with no gap (single command), watching the serial console the whole time.

First attempt (see below) targeted a directory entry's own pointer instead of a file's, and got silently healed by littlefs's own commit sequence within the same boot before it was ever read back — informative (shows the depth of the existing CRC-validated design), but not a test of this PR. Retargeting a file actually read early every boot avoided that race.

Result: the real internal assertion fired, with its genuine condition text

INFO  | Load /prefs/channels.proto
ERROR | LittleFS corruption detected: block < lfs->cfg->block_count
INFO  |
//\ E S H T /\ S T / C
ERROR | NOTE! Record critical error 13 at src/platform/stm32wl/main-stm32wl.cpp:214
INFO  | LittleFS format complete; restoring default settings
DEBUG | Filesystem files:
INFO  | Init NodeDB
ERROR | Could not open / read /prefs/nodes.proto
...
ERROR | Could not open / read /prefs/device.proto
INFO  | Install default DeviceState
ERROR | Could not open / read /prefs/config.proto
INFO  | Install default LocalConfig
ERROR | Could not open / read /prefs/module.proto
INFO  | Install default ModuleConfig
ERROR | Could not open / read /prefs/channels.proto
INFO  | Install default ChannelFile
...
INFO  | STM32WL init success

"block < lfs->cfg->block_count" is the literal stringified condition from the LFS_ASSERT this PR converts (src/platform/stm32wl/littlefs/lfs.c, lfs_cache_read()) — confirmed via disassembly that this exact check is what's compiled into the binary at that call site. Before this fix, hitting it would have hung the device forever with zero output (__wrap___assert_func is while(true);). Instead: it logged the corruption, rebooted, reformatted, and came back up fully functional (radio init succeeded, normal operation resumed) — no hang, no manual intervention.

For contrast: the "already-handled" case

Random-byte corruption of the same partition (not targeting a specific pointer) is caught earlier, by littlefs's own commit-CRC validation, and handled by the existing (pre-PR) code path:

ERROR | Error: can't decode protobuf wrong wire type
ERROR | Config decode failed - freezing identity, booting degraded (radio silent until restored)
INFO  | Install default LocalConfig

No hang either way, but this is a different (already-working) mechanism — protobuf-level validation catching garbage file contents. This PR specifically targets the narrower case where littlefs's own internal structure is corrupted in a way that still passes its CRC check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants