SD card optimized formatting

[danboid]

Recently I learned there is a special tool made by the SD Association that optimally formats SD cards to maximize their performance. The problems with this are that its not open source, you have to agree to their EULA to use it and its only available for Windows 7+ and macOS 10.7+.

Is there any chance this feature could be added to mkfs.vfat (and mkfs.exfat) or do you know of a existing open source implementation of such a tool?

https://www.sdcard.org/downloads/formatter/

[pali]

mkfs.vfat can be extended to any new functionality. Please describe exactly what is not supported and test case to verify it.

[danboid]

Hi @pali

Apologies for the long delay in replying, but it was only today that I found out exactly how to best format an SD card under Linux for best performance.

I know more now because the sdcard.org formatting tool has recently had a Linux version released ( https://www.sdcard.org/downloads/sd-memory-card-formatter-for-linux/ ) and that led to this thread on Hacker News:

https://news.ycombinator.com/item?id=35610243

From those comments, I found this post very helpful:

dmitrygr 2 hours ago | prev | next [–]

So much confusion in this discussion. Let me try to clarify (all of this is very very simplified).

1. NAND in SD cards has page size (minimum writeable piece) something like 4kB or 8KB, or maybe even 64KB. But as per spec SD cards must allow writes at 512 byte granularity. Aligning your file system clusters with these flash pages will allow SIGNIFICANTLY better performance and lower wear. How? Well in the CSD register (accessible using SD commands), the NAND geometry is given, so all it takes is reading that and creating a file system that is properly aligned.

2. Why not just mkfs.vfat? You can, but unless you do some of the above work yourself and some math, you’re unlikely to get the alignment right and will suffer because of it, especially on small or random writes. You CAN do it yourself. Just takes a little work.

3. What about my gopro, my phone, or $DEVICE-X that can format SD cards? Well, most device firmwares DO indeed do this correctly as the SD spec advises to do.

4. Why doesn’t the card just do it in firmware all by itself? Curiously sony memory stick does this. Why not sd? There isn’t a command in the spec for that and adding it now is too late - nobody will use it. Thus we have this tool and ones like it.

5. What is this about the secure area? The tool just says it doesn’t touch it. Almost nobody does. Costs a lot of money to license the SD security stuff. Nobody uses it. It was meant for securing MP3s on cards. Secure area exists on all cards (yes you are paying for flash you cannot use). Luckily it isn’t big. Special commands are used to access it. This tool and basically any device you’ll ever encounter in your everyday life do not use them. Lacking those commands, it is as if the secure area doesn’t exist.

6. Why dont windows/Linux/macOS do it right themselves? My guess? Separation of concerns. The formatting code there only formats a partition. For proper alignment you may need to start the partition on the right block as well. Simple example. Say in your fictional card, we have 4KB pages. That’s 8 512-byte sectors. So we want our first FAT cluster to start at a multiple of 8 (and be a multiple of sectors in size). Say we work out that our various FS Structures need 17 sectors. That means that we need the partition to start at sector 7. Thus we make the MBR do so. Wait! You might say that FAT already has “resevered sectors” value we can use for this. No MBR touching needed. Well, you’d be surprised how many devices/firmwares break with nonstandard values there. Also, what makes you so sure that mkfs.vfat can even find out how many sectors into /dev/sdb, /dev/sdb1 starts, to do the math right? For that matter, what makes you think it would even know to check for SD-card-ness of /dev/sdb? 

However this is the link most relevant to this feature request:

https://web.archive.org/web/20230405090708/https://3gfp.com/wp/2014/07/formatting-sd-cards-for-speed-and-lifetime/

It explains how to calculate the reserved sectors and where the data partition should begin for optimal performance.

It would be good if mkdosfs (and mkfs.vfat too?) introduced a new option to automate this process to calculate the partition offset for you.

[pali]

Just to avoid confusion, the tool is named mkfs.fat, previously it was named mkdosfs and for compatibility with other tools, install script creates symlink for mkdosfs and also for mkfs.msdos and mkfs.vfat. The invoked name of the tool is not used for anything.

mkfs.fat already aligns formatted sectors to chosen cluster size (unless filesystem is not too small). Also mkfs.fat already choose C/H/S geometry based on SD Card Part 2 File System Specification. mkfs.fat reads from OS offset of the partition from beginning of disk / sd card for proper filling of "hidden sector" field, but does not use it for aligning (or filling reserved sector). This could be an improvement in this area. But I'm not sure if it is really needed.

Normally it is enough to create MBR partition on SD card which starts at 4MB offset from the beginning. 4MB for sure is aligned to any possible NAND erase size. When partition size is larger than 8GB then mkfs.fat formats it to FAT32 with 8kB cluster size. This should be already fine.

I will comments for your points below. Based on my above comments, I think that mkfs.fat do not need other SD specific stuff. There is an area for improving cluster size selection (which also affects Fat Allocation Table size and therefore total available space for data), C/H/S geometry parameters to align with MBR (needed for some devices) or erasing/discarding sectors during formatting. Anyway if there are any other stuff which you think that needs to be handled, please let me know. But mostly performance is impacted by the writer - kernel vfat.ko driver, not formatter.

1. NAND in SD cards has page size (minimum writeable piece) something like 4kB or 8KB, or maybe even 64KB. But as per spec SD cards must allow writes at 512 byte granularity. Aligning your file system clusters with these flash pages will allow SIGNIFICANTLY better performance and lower wear. How? Well in the CSD register (accessible using SD commands), the NAND geometry is given, so all it takes is reading that and creating a file system that is properly aligned.

Access to CSD registers requires putting SD card to the SDHCI (or some mmc-compatible) controller which is connected to processor. Mostly available for industrial and embedded hardware, not common on x86 computers or laptops. But I have some laptops with SD card reader in form of PCIe based SDHCI controller, which allows it. USB mass-storage protocol (used in most SD card readers) does not have ability to access SDHCI controller inside, so no access to CSD registers.

To check if your SD card reader is not USB based and support access to CSD registers, just look if your SD card is detected as real mmc block device /dev/mmcblk0 and not as generic (scsi) device /dev/sda.

4. Why doesn’t the card just do it in firmware all by itself? Curiously sony memory stick does this. Why not sd? There isn’t a command in the spec for that and adding it now is too late - nobody will use it. Thus we have this tool and ones like it.

mmc specifications are extending and new useful things are being added and then they are also used... So this is not an argument not to add it. Instead it really does not make sense if card firmware reformats card to some old fashion filesystem like (ex)FAT. The only purpose of FAT usage in these days is compatibility with MS systems and other old system. It is not filesystem designed for performance nor for flash storage.

5. What is this about the secure area? The tool just says it doesn’t touch it.

This card area is not accessible by standard commands. It is not part of the visible block device in system. You need to use different mmc cmds for accessing it, meaning you need to talk with SDHCI controller (or other mmc compatible). Also I think that Linux kernel still does not have implemented support for it and neither exported ioctls from /dev/mmcblk... But I do not remember exact details. So no tool normally touches it.

But Linux has already support for accessing eMMC boot partitions and those are exported as another block devices in /dev/. And user can use it like any other block device.

[Artoria2e5]

I just had the misfortune of reading the "SD Specifications Part 2: File System Specification" too, so some comments here.

OS offset... This could be an improvement in this area. But I'm not sure if it is really needed.

This honestly is all that needs for me consider it "SD-optimized" or whatever. Presumably we do it like exfatprogs and accept a BU size, so the program chooses a reserve size that aligns to BU size (when combined with offset).

Normally it is enough to create MBR partition on SD card which starts at 4MB offset from the beginning.

Pedants might point to the recommended way for FAT12/16, which is to move the partition (not the reserve count) to get alignment. Don't see how that's better than using the reserve count and it's not like mkfs does partitioning anyways.

cluster size selection

The choice on the SD side is different for 1/16 vs 32 anyways, so shrug.

[anzz1]

@danboid has definitely touched an important and oft-misunderstood issue of using proper allocation unit sizes for flash memory.
The obvious issue here is that having a FAT filesystem on flash memory storing any meaningful amount of data would've been an incomprehensible thought back in 1977 at the time of the format's invention.

That is pretty much the problem of all the most popular filesystems in use today, like ext/NTFS. There was no notion of the concept of write-erase cycles at the time and thus formats designed to run on spinning disks are wholly incompatible with flash memory and have undesirable qualities. However, in the case of SSD's this problem was largely bypassed by implementing RAM caches and smart controllers employing smart algorithms which mitigate the disadvantages of using legacy disk formats on a medium they were never intended for.^[1]

However, SD cards do not do this. At most they have simple wear-leveling mechanisms but that's it. A lot is left to user, to understand the decisions regarding which filesystem to use and with which parameters. This is where everything falls off a cliff since most people do not understand any of it. Can't blame them either, since good information is hard to come by. Most of the information related to this is obsolete and simply incorrect and that information ends up being parroted everywhere in a never-ending cycle. When the first search results on Google lead to incorrect answers, most upvoted answers on sites like Stack Overflow / Serverfault^[2] / SuperUser^[3] are wrong, the official spec sheet from SD association is poorly written and concepts aren't properly explained, and even the wikipedia page for FAT^[4] is contradicting itself and factually incorrect, it's hard to blame anyone for not having the correct information. It also doesn't help that the concepts of "sector size" (hardware, disk) and "cluster size" (software, filesystem) are often used interchangably and incorrectly. Then there are terms like "allocation unit size", "block size" thrown into the mix which mean those same two concepts but only help in adding to the confusion.

1. NAND in SD cards has page size (minimum writeable piece) something like 4kB or 8KB, or maybe even 64KB. But as per spec SD cards must allow writes at 512 byte granularity. Aligning your file system clusters with these flash pages will allow SIGNIFICANTLY better performance and lower wear. How? Well in the CSD register (accessible using SD commands), the NAND geometry is given, so all it takes is reading that and creating a file system that is properly aligned.

Answers like these instantly give away that the user doesn't know what they're talking about. "something like" , "or maybe even" without any further explanation. That's the problem of obsolote information, as what is the user is saying has been historically correct, but only until 2006 when SDHC specification was released along with CSD 2.0. But that answer was given in 2023, not in 2005.

What the poster, and also pretty much 9/10 of answers found online doesn't understand, is that the SECTOR_SIZE value is largely obsolete and has been for a long time now. Clues to this are often buried in the footnotes though, like in this example Transcend datasheet, page 12^[5]

Historically, the minimum erasable block size and thus the correct allocation unit size for a SD card was WRITE_BL_LEN bytes * SECTOR_SIZE. While SD card spec allows for different sector sizes(WRITE_BL_LEN / READ_BL_LEN), I've never seen this be anything else than 512. So the assumption of 512*erase block size(SECTOR_SIZE) was correct, two decades ago. The current spec still allows using this configuration, but when the "new" (from 2006) parameter of ERASE_BLK_EN = 1, the SECTOR_SIZE means nothing and WRITE_BL_EN is the sector size, erase block size and write block size all at once.^[6][7]. SDXC being released in 2009 and most of all cards being SDXC for well over a decade, the assumptions of a "correct" block size based on a Microsoft's FAT32 specification from Dec 6, 2000^[8], no longer hold true.

Therefore the only correct way to get the proper allocation unit size for the SD card would be to get the data from the CSD, which probably isn't possible using run-of-the-mill USB SD card readers. However, if you happen to have a SD controller and mounted the SD card as a block device in linux, you can do this:

cat /sys/block/mmcblk0/device/csd

Enter that number to this tool [9] and press "Decode CSD"

If your ERASE_BLK_EN is 0x1 , then your default allocation size should be the value in WRITE_BL_LEN.
If your ERASE_BLK_EN is 0x0 , then your default allocation size should be the value in SECTOR_SIZE (the tool automatically calculates the WRITE_BL_LEN*SECTOR_SIZE bytes)

That being said, I haven't run into a single SDXC card where the ERASE_BLK_EN wasn't 1 and WRITE_BL_LEN wasn't 512.
So I'd say a safe bet for the block size on any SDXC card manufactured in the last decade would be 512.

I am not saying that the default values should be changed though, as they've been set at that for two decades now without changes in Windows(format) and Linux(dosfstools). So on a widely used tools like this, I wouldn't go on changing the defaults even if it was warranted as it could break people's workflows. And also the old block size assumptions are valid for the old base SD spec.

What I'm saying though that on users' part you should be mindful of this. The old wisdom of larger block size for larger files still stands, as more blocks means more filesystem overhead. So if your use case is only large files, like a digital camera taking high resolution photos/video and nothing else, using a large allocation unit size would be the best bet.

In every other case though, you would most definitely want to use the allocation unit size which matches the SD card, otherwise every single write and erase would be multiplied and wear out your card exponentially faster. It would be easy to say to always use 512, but FAT32 poses a hard limit of maximum sector count of (2^28-1).

That would mean that you can't use the optimal 512 on larger cards.
The most optimal values achievable on FAT32, with mkfs.fat -S 512 -s N

Card capacity	Allocation unit size	-s N
<= 128 GB	512	1
<= 256 GB	1024	2
<= 512 GB	2048	4
<= 1 TB	4096	8
<= 2 TB	8192	16
<= 4 TB	16384	32
<= 8 TB	32768	64
<= 16 TB	65536	128

Even after all these decades, FAT32 is still viable for a simple filesystem. It lacks journaling and fancy features like file permissions and attributes which you wouldn't want to use on a SD card anyway. It also doesn't have any mitigations for fragmentation, which is a good thing since (de)fragmentation was a thing of spinning disks and you wouldn't want to go near that on any flash memory. Simplicity also makes it fast, easy to implement and causes very little unnecessary wear unlike many other filesystems. However, the lack of journaling makes it very prone to failure when power is cut abruptly, which obviously can happen a lot in many applications of the SD card, like battery powered mobile devices.

The hard limit of sectors is also unfortunate since it doesn't allow optimal allocation size, reducing performance and lifespan. Also the constant updating of the file allocation tables themselves pose a wear problem, but its nowhere near as bad as on something like ext or NTFS.

Solution to all these problems has existed since 2012, f2fs^[10], which is specifically designed for commodity flash storage. However unfortunately the mainstream support for it is still pretty much nonexistent, as at least according to online information it still isn't included by default in most Linux distributions, never mind Windows or macOS. The mainline linux kernel itself has supported it since 2012 though, from v3.8 onwards.

References

[1] https://icrontic.com/article/how_ssds_work
[2] https://serverfault.com/questions/7531/would-ssd-drives-benefit-from-a-non-default-allocation-unit-size
[3] https://superuser.com/questions/1151976/why-is-exfats-default-allocation-size-in-windows-so-high
[4] https://en.wikipedia.org/wiki/File_Allocation_Table#Maximal_sizes
[5] http://transcend-info.com/products/images/modelpic/539/SDHC10I_datasheet.pdf

[6] https://ebics.net/stm32s-sd-card-internal-structure-sketch/
[7] https://goughlui.com/2014/01/03/project-read-collect-decode-sd-card-csd-register-data/
[8] https://github.com/gaofeng/EffiLib/raw/master/fatfs/fatgen103.doc

[9] https://gurumeditation.org/1342/sd-memory-card-register-decoder/

[10] https://lwn.net/Articles/518718/

[Fry-kun]

FWIW, here's CSD from an older 2G microSD card I have: 007f00325b5a83b1f6dbff8716800001

ERASE_BLK_EN	0x1
WRITE_BL_LEN	0xa == 1024 bytes

never say never I guess?

Edit: oops, guess this isn't SDXC 😝

[anzz1]

Good information there @Fry-kun, so that shows that 512 bytes isn't a rule, and cards which have other than 512B sector size do exist in the wild. So that'd be an SDHC, CSD2.0 card, right? Any idea how old it is?

I guess the assumption of all SDXC cards in the wild having 512 byte sectors hasn't been proven false, yet.

But ultimately it's needed to read the CSD to be certain of specs of any given card.