Feature Request: exFAT Support

[EdwinDavis]

Interested to learn about plans to restore exFAT support.

Were any issues encountered when using SdFat-Beta from Bill Greiman? I notice it was removed in 78a7039

[PaulZC]

Hi @EdwinDavis ,

Yes, originally we were using SdFat-Beta to provide support for exFAT, but we found it wasn't completely reliable. Logging would stop randomly after a few hours. We went back to standard SdFat and the issues went away. We do plan to reinstate SdFat-Beta but we'll need to find and cure the cause of the earlier problems - which involves some very low level diagnostics and leaving the system logging for hours...

I'm going to leave this issue open, just as a placeholder so we don't forget about exFAT support. But please don't expect it to be resolved soon!

Best wishes,

Paul

[adamgarbo]

Hey @PaulZC,

Just a quick update that it looks like Bill has updated SdFat to v2.0.0, which now supports exFAT: https://github.com/greiman/SdFat

Curious if the issues you were encountering previously were addressed.

Cheers,
Adam

[nseidle]

I'm not entirely comfortable 'fixing' something that isn't broken. I worry we'll find gremlins in the field causing failed or corrupt logs. What is the need for exFAT? >32GB card support or is there other features?

[adamgarbo]

Hi @nseidle,

I believe the biggest draw to users is having support for >32 GB. However, this was more of a heads up as Bill has brought the changes he introduced in SdFat-beta to the core SdFat library, which is now available on the Arduino IDE. Future OLA firmware versions will need to make sure that SdFat is downgraded to v1.1.4 to avoid the gremlins previously experienced.

In the meantime, I'm happy to compile a custom OLA binary with SdFat 2.0 and kick the tires to see how it works.

Cheers,
Adam

[adamgarbo]

Hi folks,

I've just completed a 24-hour test with the OLA and v2.0.4 of the SdFat library.

In the OLA firmware, I replaced all instances of SdFat and SdFile with the SdFs and FsFile classes, which simultaneously supports FAT16/FAT32 and ExFat. I used a 64 GB SanDisk Extreme microSD card formatted as exFat and had the OLA log its onboard IMU data, as well as datetime, voltage, measurement count, etc. at 10 Hz.

From the graphs below, it's possible to observe that the logging did not appear to encounter any issues while configured as such. Over 800,000 samples were logged (92 MB file) and the difference calculations on the counter variable indicate that not a single SD write was missed.

Now that I've freed up a LiPo battery, I can connect the OLA to one of the Qwiic sensors I have laying about and run another test, perhaps this time for a week, in order to test the stability of the new SdFat library and the SdFs and FsFile classes. If issues are encountered, we should probably bring it to the attention of Bill and get his thoughts on the matter.

Cheers,
Adam

[nseidle]

Really nice Adam. Thank you! Please continue to run lots of system fresh restarts. The failures I would see happened within the first hour. It's reassuring to see a 24 hour test. If you can show 20-30 successful one hour tests, I would be comfortable migrating.

[adamgarbo]

Hi @nseidle,

I've just finished conducting two dozen individual logging tests of the OLA with the new SdFat library/classes.

The configuration was the same as before, but this time I manually reset the OLA using the reset button every hour. I also left the OLA running overnight. Based on the graphs below, you can observe there were no failures encountered or missing writes.

Overall, a good sign that the library is performing reliably. I suspect the main difference here is that I'm using the official v2.0 release, whereas we were previously using the SdFat-Beta, which likely still had a number of bugs present.

I'll leave the decision to migrate up to you and @PaulZC! I just like graphs! 😉

Cheers,
Adam

[PaulZC]

Nice! Thank you @adamgarbo! I think that gives us the confidence to have a go at migrating to v2.0 of SdFat.
All the best,
Paul

[PaulZC]

Hi @EdwinDavis ,

Thank you for your patience. (This has taken us a while!) Version 1.11 of the firmware supports exFAT SD cards. Internally we are using version 2.0.6 of Bill Greiman's SdFat library. Please follow the upgrade instructions to update your firmware. I'm able to swap seamlessly between FAT32 and exFAT SD cards for logging, no reconfiguration necessary.

If you're interested, the critical new code is here:

OpenLog_Artemis/Firmware/OpenLog_Artemis/OpenLog_Artemis.ino

Lines 173 to 194 in 1859ea6

	#include <SdFat.h> //SdFat v2.0.6 by Bill Greiman: http://librarymanager/All#SdFat_exFAT
	#define SD_FAT_TYPE 3 // SD_FAT_TYPE = 0 for SdFat/File, 1 for FAT16/FAT32, 2 for exFAT, 3 for FAT16/FAT32 and exFAT.
	#define SD_CONFIG SdSpiConfig(PIN_MICROSD_CHIP_SELECT, SHARED_SPI, SD_SCK_MHZ(24)) // 24MHz
	#if SD_FAT_TYPE == 1
	SdFat32 sd;
	File32 sensorDataFile; //File that all sensor data is written to
	File32 serialDataFile; //File that all incoming serial data is written to
	#elif SD_FAT_TYPE == 2
	SdExFat sd;
	ExFile sensorDataFile; //File that all sensor data is written to
	ExFile serialDataFile; //File that all incoming serial data is written to
	#elif SD_FAT_TYPE == 3
	SdFs sd;
	FsFile sensorDataFile; //File that all sensor data is written to
	FsFile serialDataFile; //File that all incoming serial data is written to
	#else // SD_FAT_TYPE == 0
	SdFat sd;
	File sensorDataFile; //File that all sensor data is written to
	File serialDataFile; //File that all incoming serial data is written to
	#endif // SD_FAT_TYPE

I'm closing this issue as I believe it is fully resolved, but please reopen if you see any issues with the new firmware.
Best wishes,
Paul