Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exFat corruption - better support for exFat file system #161

Closed
matteopiccioni opened this issue Aug 30, 2018 · 21 comments
Closed

exFat corruption - better support for exFat file system #161

matteopiccioni opened this issue Aug 30, 2018 · 21 comments

Comments

@matteopiccioni
Copy link

@matteopiccioni matteopiccioni commented Aug 30, 2018

Hello, I hope this is the right place to report the issues.
With microSD exFat filesystem cards, we have consistent corruption issues when using homebrew apps (pfba, retronx, etc)
Thanks

@fincs
Copy link
Contributor

@fincs fincs commented Aug 30, 2018

This is not a problem with libnx. This is a problem with Nintendo's OS.

@matteopiccioni
Copy link
Author

@matteopiccioni matteopiccioni commented Aug 30, 2018

ok, is strange that the problem happens only with homebrew
Games never causes sd corruption
Hope that next Nintendo firmware could give more exFat stability
Thanks

@CTCaer
Copy link
Contributor

@CTCaer CTCaer commented Aug 30, 2018

Games do not force close like that or write stuff in the sd.

What version fw you have? I need it for my collection.

@matteopiccioni
Copy link
Author

@matteopiccioni matteopiccioni commented Aug 30, 2018

Ok, I understand.
I have fw 3.0.1 with exFat support

@profi200
Copy link
Contributor

@profi200 profi200 commented Sep 4, 2018

What kind of corruption do you get?

Trying to explicitly flush files before closing could maybe help. I dunno.

@jarulo
Copy link
Contributor

@jarulo jarulo commented Sep 5, 2018

Also adding my comment here to add that this is a serious bug affecting many homebrew users.
So can confirm this bug is real and very annoying but unknown how to reproduce accurately

@jarulo
Copy link
Contributor

@jarulo jarulo commented Sep 5, 2018

Some rumor is floating around that it is due to appending an existing file on exFAT, but have not gotten around to making a test case for this

@matteopiccioni
Copy link
Author

@matteopiccioni matteopiccioni commented Sep 10, 2018

I'd like to add that I had the problem using MacOS (I dont know if its important or not)
Using MacOS I need to remove (and add) the archive bit on files on the card in this way:
sudo chflags -R arch /Volumes/SDVOLUME/
sudo chflags -R noarch /Volumes/SDVOLUME/Nintendo/
as I read here

@CTCaer
Copy link
Contributor

@CTCaer CTCaer commented Sep 10, 2018

@matteopiccioni
The archive bit has nothing to do with the corruption. Only with missing files/folders inside homebrew or HOS, which otherwise exist in the sd card.

Any rumor about this is false.

The bug is in Nintendo's exfat driver. It constantly changes folders/files even on reads. And because it never syncs properly, on a hang/force quit/force reboot/power off, the handles are lost and the files/folders become missing from the file allocation table. Because exFAT does not have a 2nd FAT like fat32, these file/folders cannot be recovered.

@fincs
Copy link
Contributor

@fincs fincs commented Sep 10, 2018

Thank you for clearing this up. I'm closing this issue.

@fincs fincs closed this as completed Sep 10, 2018
@nopjmp
Copy link

@nopjmp nopjmp commented Sep 11, 2018

@CTCaer while the exFAT driver might be bad, fsdevCommitDevice has a comment attached to it from @yellows8 stating that this needs to be used after each file close.

/// Uses fsFsCommit() with the specified device. This must be used after any savedata-write operations(not just file-write). This should be used after each file-close where file-writing was done.

I have not be able to reproduce the corruption through my testing with a test app I made that writes out a 4kb file and hashes it on my specific card. A few others can reliable reproduce it with the same program randomly executing the predefined actions I've created.

Unless you know specifically why that would be the case, I believe we need to look into this more and closing the issue is premature.

@CTCaer
Copy link
Contributor

@CTCaer CTCaer commented Sep 11, 2018

fsdevCommitDevice is for savedata containers. Not for sd card and normal files.

@yellows8
Copy link
Contributor

@yellows8 yellows8 commented Sep 11, 2018

Does this still happen with 6.0.0 title-819-firmware running?

@nopjmp
Copy link

@nopjmp nopjmp commented Sep 11, 2018

I haven't tried yet but I'll ask them to try.

@profi200
Copy link
Contributor

@profi200 profi200 commented Sep 15, 2018

Going by the explanation above from @CTCaer it doesn't seem like a bug to me. It's simply caching everything for performance and if you force shutdown or it crashes that data is lost.

@nopjmp
Copy link

@nopjmp nopjmp commented Sep 15, 2018

This happens with a normal shutdown for some users. Home button more often than others. I haven't been able to replicate anything and neither has a person using 6.0. I'm waiting for release just in case since my console isn't banned yet to run 6.0. I'll report back once that happens for me and others.

@CTCaer
Copy link
Contributor

@CTCaer CTCaer commented Sep 15, 2018

It is an architecture bug. No sane OS or filesystem driver touches anything else than what the app ask it to.
They should at least close the handles to PrFILE 2 when done with setting the archive bit on a folder.
Or at least keep these to a system level cache. Not per app.

That's why it also affects normal shutdown. Cache and handles are per app/applet.

@profi200
Copy link
Contributor

@profi200 profi200 commented Sep 16, 2018

Don't they have anything in place to force sync with the microSD? If this happens even on normal shutdown then it's fucked up. They should write back everything on shutdown/reboot. How does this not break with official apps?

@D3m3rion
Copy link

@D3m3rion D3m3rion commented Mar 18, 2019

Just had my microSD corrupted. I'm on 3.0.0 (waiting till I can update without burning any fuses) and using Homebrew, no CFW.
I was playing a GBA rom that suddenly started lagging so I closed the game. At this point, there already were all sorts of corrupted files and I couldn't restart the game.
When I closed the Homebrew launcher it said my sd couldn't be read, PC says it has to be formatted.
I wonder why this happened. I assume it started lagging because of the corruption and not vice versa.

@TuxSH
Copy link
Contributor

@TuxSH TuxSH commented Mar 18, 2019

This, again, is an issue with Nintendo's FS drivers, which is super crappy, and is known to corrupt exFAT partitions.

Formatting to FAT32 works around this issue. Nintendo fragment files >= 4GB, so FAT32 is not an issue in that regard.

@fincs
Copy link
Contributor

@fincs fincs commented Mar 18, 2019

Like it has been hinted previously, this is a bug in Horizon OS itself and has nothing to do with libnx or nx-hbloader/nx-hbmenu (there's no such thing as "Homebrew Launcher" on the Switch). Even on the latest version (7.0.1 at the time of writing), exFAT is still, ahem, hosed on HOS and the only advice we can give is to stay away completely from exFAT cards and exFAT firmware.

You can just set up autorcm with hekate (and also make a nand backup); and proceed to use vanilla Atmosphère with the standard system update function to get on latest version with homebrew support, if that's what you're interested in.

Locking this thread due to necroposting.

@switchbrew switchbrew locked as off-topic and limited conversation to collaborators Mar 18, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants