FAT

FAT

About

FAT is the "File Allocation Table" file system, popular in many older operating systems, particularly MS-DOS. It was originally developed in 1977 for use on floppy disks, and uses a variety of allocation tables sizes (typically 12, 16 or 32bit, which we then call "FAT12", "FAT16" or "FAT32"). These limitations will have impacts on the number of sectors and size of clusters that can be addressed, and from that the size of the partition and/or amount of data it can hold.

For more technical details on FAT, see the Wikipedia article:

• https://en.wikipedia.org/wiki/File_Allocation_Table

Reasons to use FAT with RetroNAS

As RetroNAS tends to expose data via higher level protocols that don't need the specific features of FAT, typically FAT of any kind is NOT recommended. At modern levels of data storage, FAT is prone to corruption on very large file systems, and requires lengthy manual scanning on unclean shutdown, known as "fsck" (File System Check) or "chkdsk.exe" (Check Disk) in various operating systems.

For a good quality, reliable, modern filesystem to use with higher level protocols like Samba, FTP, HTTP, FSP, or other services, see BtrFS.

Historically FAT has had good quality open source libraries available for it which has made it popular on open source and embedded projects (slightly more modern alternatives like exFAT had legally encumbered licenses which saw several lawsuites from Microsoft against open source users, however this has changed in recent years).

There is, however, a single reason to use FAT with RetroNAS, and that is for the EtherDFS project. EtherDFS is an extremely low level, lightweight protocol. It uses layer 2 networking (no awareness of IP addressing) to share information (typically a Linux server talking to an MS-DOS client), and due to it's very lightweight nature requires a file system that mimics what MS-DOS expects, such as FAT. A specific note on EtherDFS is that MS-DOS+EtherDFS expect filenames in "8.3" format (i.e.: 8 characters for the filename portion, 3 characters for the file extension portion, and a period/dot in the middle). More information is provided in the following sections.

How to make FAT file system

Creating a file system can be done either via the Cockpit GUI, or the command line. Please be careful when creating file systems - this is often called "formatting" a file system, and will destroy all of the data on that file system. If you point to the wrong device, you can irrecoverably wipe data in use, so always ensure to double check device names and IDs.

Graphical method

The graphical method involves using Cockpit. See that page for a comprehensive guide, including a video on how to use it. At time of format, the "VFAT" file system is the option you want to choose, regardless of how you will mount the file system later.

Command line method

Linux provides the "mkfs" family of tools to create file systems and format disks. "mkfs.vfat", "mkfs.fat" and "mkfs.msdos" are all the same tool (simply a symlink pointing them to the same binary). Using these on the command line, you can use the following options as an example:

mkfs.vfat -F 32 -n NAME /dev/sdx1

Where:

• "-F" specifies the FAT size. Typically 12, 16 or 32. 32 is recommended even if your client OS can't necessarily address FAT32 (for example, MS-DOS), however that's fine as this isn't the limitation of EtherDFS.
• "-n" specifies the file system or volume name. This is recommended to be 8 characters or fewer, and all UPPER CASE.
• "/dev/sdx1" specifies the device and partition ID you want to format. Take great care here. This can be a real partition on a physical disk, a MD-RAID volume, an LVM logical volume, a loopback file, or many other options. Examples of these will follow.

Mounting the FAT volume

Linux offers several methods to mount FAT volumes. Typically the "vfat" mount type (i.e.: mount -t vfat /dev/sdx1 /path/to/mountpoint) is used, which allows advanced FAT features such as long file names to be used. Specific to EtherDFS, it is recommended instead to use the "msdos" (i.e. mount -t msdos /dev/sdx1 /path/to/mountpoint) method. This will use standard MS-DOS filename truncation, where long filenames will be shortened, and conflicting long filenames (e.g.: "longfilename1.txt" and "longfilename2.txt") will be shortened and numbered with tilde characters (i.e.: appearing as "LONGFI~1.TXT" and "LONGFI~2.TXT").

Mounting the file system can be done similarly in Linux's /etc/fstab (File System TABle) that mounts file systems at boot. Specify "msdos" as the mount type here too.

There is no restriction on how the file system was originally formatted or mounted. You can re-mount at a later date as type "msdos" without losing data. Just take note of the automatic file name shortening feature, and that it's not always obvious how the file system will choose to number the files "~1", "~2" and so on.

Advanced usage

Here's an example of advanced usage for EtherDFS. Let's suppose we have a 10TB USB3 spindle hard disk attached to a Raspberry Pi 4, and have installed Samba on it to share files with both modern Windows 11 clients as well as retro Windows 95 clients. We are going to assume the default RetroNAS path of /data/retronas and a SMB share named \\retrosmb\retronas exposing that top level, and we are going to assume that the file system has been formatted as BtrFS.

Installing EtherDFS will create a real folder named /data/retronas/dos, which can be accessed via the SMB path \\retrosmb\retronas\dos. However by default this file system is the same as the entire file system for the 10TB USB3 disk.

Our hypothetical /etc/fstab file in Linux will contain a line looking something like:

/dev/sda1  /data/retronas  btrfs  noatime,compress=lzo  0 0

Where /dev/sda1 is our drive ID in Linux. This might be referenced by UUID instead and point to an ext4 volume, so it could look like:

UUID=03ae24f3-47f0-406a-9218-fc88be809a7e  /data/retronas  ext4   defaults  0 0

Or something to these effects.

Let's create an empty file that we will format and loopback mount. This file will contain a FAT32 filesystem within it, but exist as a file on top of our filesystem. We'll create this as a sparse file - i.e.: it won't take up space until files exist within it. In addition, if this file lives on top of a BtrFS volume, it will take advantage of BtrFS's inline compression and block level checksumming of the physical disk below it.

To create the file, we use the Linux dd command with some special options (this is all running as the "pi" user, or whatever user you have configured as your RetroNAS user):

mkdir /data/retronas/vhd
cd /data/retronas/vhd
dd if=/dev/zero of=/data/retronas/vhd/etherdfs.img bs=1G count=0 seek=8192

This creates an "8TB" sparse file, although it won't use up actual space until data is placed within it. Verifying:

ls -lah etherdfs.img
-rw-r--r-- 1 pi pi 8.0T Apr 17 12:50 etherdfs.img

du -h --apparent-size etherdfs.img
8.0T  etherdfs.img

du -h etherdfs.img
0       etherdfs.img

We can see the file takes up 0 bytes of space, but appears as if there's 8TB of space taken up.

Let's format the file, and test the loopback mount

sudo mkfs.vfat -F 32 etherdfs.img

Which produces output:

mkfs.fat 4.1 (2017-01-24)
Warning: target too large, space at end will be left unused

Notice we get a warning about the maximum size of FAT32 by default. Let's mount it (assuming you have /data/retronas/dos as an available folder, created by the EtherDFS installer by default. If not, create it first with mkdir):

sudo mount -t msdos -o loop,uid=pi,gid=pi /data/retronas/vhd/etherdfs.img /data/retronas/dos
df -hT /data/retronas/dos

Which produces output:

Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/loop0     msdos  2.0T   32K  2.0T   1% /data/retronas/dos

Notice the available space is only 2TB, which is the default for FAT32 formatted with 512 byte logical sector sizes. We can change that all the way up to 4KB (4096 byte) logical sector sizes, giving us a maximum volume size of 16TB (the absolute maximum upper bounds of a FAT32 file system). We're using an 8TB sparse file, so we want to use the full 8TB of space.

Let's do that:

cd /data/retronas/vhd
sudo umount /data/retronas/dos
mkfs.vfat -F 32 -S 4096 etherdfs.img
sudo mount -t msdos -o loop,uid=pi,gid=pi,noatime etherdfs.img /data/retronas/dos
df -hT /data/retronas/dos

And we'll see output:

Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/loop0     msdos  8.0T  256K  8.0T   1% /data/retronas/dos

Our image now appears as 8TB of usable space. We can make this permanent by adding a line in our /etc/fstab AFTER our top level /data/retronas entry:

/data/retronas/vhd/etherdfs.img  /data/retronas/dos  msdos  loop,noatime,uid=pi,gid=pi  0 0 

This will mount that file system on boot. EtherDFS is configured to share out of the dos directory under the RetroNAS top level directory, and can now use the FAT32 file system there. If the underlying file system is BtrFS configured with lzo (best for RPi or retro x86 level hardware) or zstd (best for modern x86 level hardware) compression, the entire image file and contents will be transparently compressed in realtime. We add the "noatime" flag to stop access time updates (i.e.: changes to the internal file system on reads), and set the UID/GID to the "pi" user (our default RetroNAS user).

This path can be accessed via Samba and SMB/CIFS file sharing from the top level directory. We've used the "msdos" mount type to ensure long file names are truncated to keep MS-DOS happy.

Because we used a sparse file system, we won't take up the full 8TB of space immediately, meaning that we can share the space fairly with the underlying file system.

Reclaiming space

As we fill up the sparse file, space will be taken up. Let's create some uncompressible files with random data and see the effect on our sparse image file:

du -h /data/retronas/vhd/etherdfs.img 
257M    /data/retronas/vhd/etherdfs.img

dd if=/dev/urandom of=/data/retronas/dos/rand1 bs=1M count=100
dd if=/dev/urandom of=/data/retronas/dos/rand2 bs=1M count=100
dd if=/dev/urandom of=/data/retronas/dos/rand3 bs=1M count=100
sync

du -shx /data/retronas/vhd/etherdfs.img 
557M    /data/retronas/vhd/etherdfs.img

Our empty image file allocated around 256MB of data+metadata as part of the FAT formatting. We add 3 files that are 100MB of random (uncomressible) data, "sync" (force write all cached data to disk) and re-check the image file. It's grown around 300MB as expected. Let's delete that data and see what happens:

rm /data/retronas/dos/rand*
du -shx /data/retronas/vhd/etherdfs.img 
557M    /data/retronas/vhd/etherdfs.img

The files were deleted, but we haven't reclaimed any space. We'll use the "fstrim" command, which applies the "TRIM" disk operation to reclaim space on the actual disk surface. This same command can be used to ensure flash (SSD, NVME, etc) media reclaims sector space, and it works with image files, loopback mounts, and other virtual storage too.

sudo fstrim -v /data/retronas/dos
/data/retronas/dos: 8 TiB (8795823800320 bytes) trimmed

sync

du -h /data/retronas/vhd/etherdfs.img 
257M    /data/retronas/vhd/etherdfs.img

du -h --apparent-size /data/retronas/vhd/etherdfs.img 
8.0T    /data/retronas/vhd/etherdfs.img

df -hT /data/retronas/dos/
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/loop0     msdos  8.0T  256K  8.0T   1% /data/retronas/dos

Our image / virtual hard disk has now returned to it's minimum 256MB size, despite still appearing like an 8TB FAT file system available to EtherDFS.

For more information on the trim command/function, see this Wikipedia article:

• https://en.wikipedia.org/wiki/Trim_(computing)

A good option from here is to run fstrim as a once-per-day command via CRON (not advised to run constantly/hourly due to its speed impact on IO as it runs). For example, creating a file named /etc/cron.d/dostrim with the contents:

30 05 * * *   root   /usr/sbin/fstrim /data/retronas/dos

Will run fstrim at 5:30am every morning, reclaiming any used space from deleted data prior.