nuttx - statfs() might lead to halt

[BazookaJoe1900]

note-this is duplication to issue i Opened on PX4/NuttX (original Issue). I am not sure where to put that and one can be removed.

Describe the bug
I added mavlink message, that checks the SD card status periodically (in contrast to current implementation STORAGE_INFORMATION, that need to be asked by the ground).
during checking my message, I tested what will happen if I remove the SD card during operation. doing that caused the mavlink thread to stop.

first, do you consider that as a problem? how the system behave if SD is removed, or get malfunction during flight.
I only checked that above scenario, (added periodic check mavlink message). but i guess that other methods that required the SD are much more critical, for example reading mission on commander....

I think that it related to the fact that reading the status of fat32 is done by fat_statfs(), that waits for semaphore fat_semtake(fs);

To Reproduce
Testing code can be found at:
https://github.com/BazookaJoe1900/Firmware/tree/testing-sd_removal

Steps to reproduce the behavior:

1. start logging (use 'logger on' for example)
2. remove the SD card
3. most of times, the mavlink will stop. you can all see that using top that the mavlink is on 0%

[dagar]

Yes I consider this a problem if an SD card failure can cause the mavlink module or navigator (guessing) to stop responding. Can we handle the failure gracefully in dataman?

[BazookaJoe1900]

That is not that simple. its something deeper on the OS.
@davids5, any thoughts?

[davids5]

Yes I agree it should fail with a timeout. I will look into this, but it will be after the 16 th,

[BazookaJoe1900]

Note that the timeout need to be somewhere on the writer or the other process that holding the semaphore.

[dagar]

That is not that simple. its something deeper on the OS.

I know, I mean that will be the ultimate test once the core problem is addressed. If in the mission the failure also needs to trigger a failsafe otherwise you'll get stuck on the last mission item.

[julianoes]

If in the mission the failure also needs to trigger a failsafe otherwise you'll get stuck on the last mission item.

Presumably this is all implemented. It should complain about a failure to load the mission and be in the same state as after a finished mission.

[BazookaJoe1900]

From what I seen, writing to the SD, and other operations with it doesn't has timeouts.
so other task that writing to the SD (lets say the logger) can block the access to the SD.
The blocking can be permanent if there was an error, for example, removing the SD.
so the commander, will try to access the SD, and get stack there because the writer will not free the semaphore.

[mrpollo]

I'm guessing this is dev call material, anyone here would like to lead the discussion on the next dev call Oct 16th?