Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exFat corruption - better support for exFat file system #161

Closed
matteopiccioni opened this issue Aug 30, 2018 · 21 comments

Comments

Projects
None yet
9 participants
@matteopiccioni
Copy link

commented Aug 30, 2018

Hello, I hope this is the right place to report the issues.
With microSD exFat filesystem cards, we have consistent corruption issues when using homebrew apps (pfba, retronx, etc)
Thanks

@fincs

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2018

This is not a problem with libnx. This is a problem with Nintendo's OS.

@matteopiccioni

This comment has been minimized.

Copy link
Author

commented Aug 30, 2018

ok, is strange that the problem happens only with homebrew
Games never causes sd corruption
Hope that next Nintendo firmware could give more exFat stability
Thanks

@CTCaer

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2018

Games do not force close like that or write stuff in the sd.

What version fw you have? I need it for my collection.

@matteopiccioni

This comment has been minimized.

Copy link
Author

commented Aug 30, 2018

Ok, I understand.
I have fw 3.0.1 with exFat support

@profi200

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2018

What kind of corruption do you get?

Trying to explicitly flush files before closing could maybe help. I dunno.

@jarulo

This comment has been minimized.

Copy link
Contributor

commented Sep 5, 2018

Also adding my comment here to add that this is a serious bug affecting many homebrew users.
So can confirm this bug is real and very annoying but unknown how to reproduce accurately

@jarulo

This comment has been minimized.

Copy link
Contributor

commented Sep 5, 2018

Some rumor is floating around that it is due to appending an existing file on exFAT, but have not gotten around to making a test case for this

@matteopiccioni

This comment has been minimized.

Copy link
Author

commented Sep 10, 2018

I'd like to add that I had the problem using MacOS (I dont know if its important or not)
Using MacOS I need to remove (and add) the archive bit on files on the card in this way:
sudo chflags -R arch /Volumes/SDVOLUME/
sudo chflags -R noarch /Volumes/SDVOLUME/Nintendo/
as I read here

@CTCaer

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2018

@matteopiccioni
The archive bit has nothing to do with the corruption. Only with missing files/folders inside homebrew or HOS, which otherwise exist in the sd card.

Any rumor about this is false.

The bug is in Nintendo's exfat driver. It constantly changes folders/files even on reads. And because it never syncs properly, on a hang/force quit/force reboot/power off, the handles are lost and the files/folders become missing from the file allocation table. Because exFAT does not have a 2nd FAT like fat32, these file/folders cannot be recovered.

@fincs

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2018

Thank you for clearing this up. I'm closing this issue.

@fincs fincs closed this Sep 10, 2018

@nopjmp

This comment has been minimized.

Copy link

commented Sep 11, 2018

@CTCaer while the exFAT driver might be bad, fsdevCommitDevice has a comment attached to it from @yellows8 stating that this needs to be used after each file close.

/// Uses fsFsCommit() with the specified device. This must be used after any savedata-write operations(not just file-write). This should be used after each file-close where file-writing was done.

I have not be able to reproduce the corruption through my testing with a test app I made that writes out a 4kb file and hashes it on my specific card. A few others can reliable reproduce it with the same program randomly executing the predefined actions I've created.

Unless you know specifically why that would be the case, I believe we need to look into this more and closing the issue is premature.

@CTCaer

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2018

fsdevCommitDevice is for savedata containers. Not for sd card and normal files.

@yellows8

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2018

Does this still happen with 6.0.0 title-819-firmware running?

@nopjmp

This comment has been minimized.

Copy link

commented Sep 11, 2018

I haven't tried yet but I'll ask them to try.

@profi200

This comment has been minimized.

Copy link
Contributor

commented Sep 15, 2018

Going by the explanation above from @CTCaer it doesn't seem like a bug to me. It's simply caching everything for performance and if you force shutdown or it crashes that data is lost.

@nopjmp

This comment has been minimized.

Copy link

commented Sep 15, 2018

This happens with a normal shutdown for some users. Home button more often than others. I haven't been able to replicate anything and neither has a person using 6.0. I'm waiting for release just in case since my console isn't banned yet to run 6.0. I'll report back once that happens for me and others.

@CTCaer

This comment has been minimized.

Copy link
Contributor

commented Sep 15, 2018

It is an architecture bug. No sane OS or filesystem driver touches anything else than what the app ask it to.
They should at least close the handles to PrFILE 2 when done with setting the archive bit on a folder.
Or at least keep these to a system level cache. Not per app.

That's why it also affects normal shutdown. Cache and handles are per app/applet.

@profi200

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2018

Don't they have anything in place to force sync with the microSD? If this happens even on normal shutdown then it's fucked up. They should write back everything on shutdown/reboot. How does this not break with official apps?

@D3m3rion

This comment has been minimized.

Copy link

commented Mar 18, 2019

Just had my microSD corrupted. I'm on 3.0.0 (waiting till I can update without burning any fuses) and using Homebrew, no CFW.
I was playing a GBA rom that suddenly started lagging so I closed the game. At this point, there already were all sorts of corrupted files and I couldn't restart the game.
When I closed the Homebrew launcher it said my sd couldn't be read, PC says it has to be formatted.
I wonder why this happened. I assume it started lagging because of the corruption and not vice versa.

@TuxSH

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2019

This, again, is an issue with Nintendo's FS drivers, which is super crappy, and is known to corrupt exFAT partitions.

Formatting to FAT32 works around this issue. Nintendo fragment files >= 4GB, so FAT32 is not an issue in that regard.

@fincs

This comment has been minimized.

Copy link
Contributor

commented Mar 18, 2019

Like it has been hinted previously, this is a bug in Horizon OS itself and has nothing to do with libnx or nx-hbloader/nx-hbmenu (there's no such thing as "Homebrew Launcher" on the Switch). Even on the latest version (7.0.1 at the time of writing), exFAT is still, ahem, hosed on HOS and the only advice we can give is to stay away completely from exFAT cards and exFAT firmware.

You can just set up autorcm with hekate (and also make a nand backup); and proceed to use vanilla Atmosphère with the standard system update function to get on latest version with homebrew support, if that's what you're interested in.

Locking this thread due to necroposting.

@switchbrew switchbrew locked as off topic and limited conversation to collaborators Mar 18, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.