exFat corruption - better support for exFat file system

[matteopiccioni]

Hello, I hope this is the right place to report the issues.
With microSD exFat filesystem cards, we have consistent corruption issues when using homebrew apps (pfba, retronx, etc)
Thanks

[fincs]

This is not a problem with libnx. This is a problem with Nintendo's OS.

[matteopiccioni]

ok, is strange that the problem happens only with homebrew
Games never causes sd corruption
Hope that next Nintendo firmware could give more exFat stability
Thanks

[CTCaer]

Games do not force close like that or write stuff in the sd.

What version fw you have? I need it for my collection.

[matteopiccioni]

Ok, I understand.
I have fw 3.0.1 with exFat support

[profi200]

What kind of corruption do you get?

Trying to explicitly flush files before closing could maybe help. I dunno.

[jarulo]

Also adding my comment here to add that this is a serious bug affecting many homebrew users.
So can confirm this bug is real and very annoying but unknown how to reproduce accurately

[jarulo]

Some rumor is floating around that it is due to appending an existing file on exFAT, but have not gotten around to making a test case for this

[matteopiccioni]

I'd like to add that I had the problem using MacOS (I dont know if its important or not)
Using MacOS I need to remove (and add) the archive bit on files on the card in this way:
sudo chflags -R arch /Volumes/SDVOLUME/
sudo chflags -R noarch /Volumes/SDVOLUME/Nintendo/
as I read here

[CTCaer]

@matteopiccioni
The archive bit has nothing to do with the corruption. Only with missing files/folders inside homebrew or HOS, which otherwise exist in the sd card.

Any rumor about this is false.

The bug is in Nintendo's exfat driver. It constantly changes folders/files even on reads. And because it never syncs properly, on a hang/force quit/force reboot/power off, the handles are lost and the files/folders become missing from the file allocation table. Because exFAT does not have a 2nd FAT like fat32, these file/folders cannot be recovered.

[fincs]

Thank you for clearing this up. I'm closing this issue.

[nopjmp]

@CTCaer while the exFAT driver might be bad, fsdevCommitDevice has a comment attached to it from @yellows8 stating that this needs to be used after each file close.

libnx/nx/include/switch/runtime/devices/fs_dev.h

Line 35 in aee9ba4

/// Uses fsFsCommit() with the specified device. This must be used after any savedata-write operations(not just file-write). This should be used after each file-close where file-writing was done.

I have not be able to reproduce the corruption through my testing with a test app I made that writes out a 4kb file and hashes it on my specific card. A few others can reliable reproduce it with the same program randomly executing the predefined actions I've created.

Unless you know specifically why that would be the case, I believe we need to look into this more and closing the issue is premature.

[CTCaer]

fsdevCommitDevice is for savedata containers. Not for sd card and normal files.

[yellows8]

Does this still happen with 6.0.0 title-819-firmware running?

[nopjmp]

I haven't tried yet but I'll ask them to try.

[profi200]

Going by the explanation above from @CTCaer it doesn't seem like a bug to me. It's simply caching everything for performance and if you force shutdown or it crashes that data is lost.

[nopjmp]

This happens with a normal shutdown for some users. Home button more often than others. I haven't been able to replicate anything and neither has a person using 6.0. I'm waiting for release just in case since my console isn't banned yet to run 6.0. I'll report back once that happens for me and others.

[CTCaer]

It is an architecture bug. No sane OS or filesystem driver touches anything else than what the app ask it to.
They should at least close the handles to PrFILE 2 when done with setting the archive bit on a folder.
Or at least keep these to a system level cache. Not per app.

That's why it also affects normal shutdown. Cache and handles are per app/applet.

[profi200]

Don't they have anything in place to force sync with the microSD? If this happens even on normal shutdown then it's fucked up. They should write back everything on shutdown/reboot. How does this not break with official apps?

[D3m3rion]

Just had my microSD corrupted. I'm on 3.0.0 (waiting till I can update without burning any fuses) and using Homebrew, no CFW.
I was playing a GBA rom that suddenly started lagging so I closed the game. At this point, there already were all sorts of corrupted files and I couldn't restart the game.
When I closed the Homebrew launcher it said my sd couldn't be read, PC says it has to be formatted.
I wonder why this happened. I assume it started lagging because of the corruption and not vice versa.

[TuxSH]

This, again, is an issue with Nintendo's FS drivers, which is super crappy, and is known to corrupt exFAT partitions.

Formatting to FAT32 works around this issue. Nintendo fragment files >= 4GB, so FAT32 is not an issue in that regard.

[fincs]

Like it has been hinted previously, this is a bug in Horizon OS itself and has nothing to do with libnx or nx-hbloader/nx-hbmenu (there's no such thing as "Homebrew Launcher" on the Switch). Even on the latest version (7.0.1 at the time of writing), exFAT is still, ahem, hosed on HOS and the only advice we can give is to stay away completely from exFAT cards and exFAT firmware.

You can just set up autorcm with hekate (and also make a nand backup); and proceed to use vanilla Atmosphère with the standard system update function to get on latest version with homebrew support, if that's what you're interested in.

Locking this thread due to necroposting.