How to fill leftover drive space with k32 plots
The concept of replotting from k32s to a mixture of k32, k33, and k34 to fill up drives is tedious, time consuming, and wasteful. Instead, it is possible to combine the leftover space using loopback devices and multiple device tools like Logical Volume Manager (LVM).
Nearly every drive size (other than 16tb) will have various amounts of leftover space after being filled with k32. With plots made by Mad Max and Bladebit plotters, this is a highly predictable amount due to truncated table sizes. With the official chia plotter, slight variations in plot size may make this less predictable. As a simple example, 10TB drives will end up with 91 k32 plots and 90-92GiB left over after being filled with plots when using the following format options:
# mkfs.ext4 -m 0 -O ^has_journal -T largefile4 /dev/sdx1
Let's assume we have two 10TB drives which can hold 91 plots each and have 90GiB left over. Rather than deleting some plots and replotting k33/k34 or letting that space sit there, we can instead create an .img file to take up the remaining space on the drive, mount that .img file as a loopback device, and then use LVM to concatenate all of the space contained in those loopback devices together.
Downside of loopback devices: they are not persistent after reboot, so they must be recreated after each startup which can be automated with a script.
LVM consists of three layers: Physical Volumes (PV), Volume Groups (VG), and Logical Volumes (LV).
Physical Volumes are block devices that have been passed through to LVM for use.
Volume Groups are groups of passed through physical volumes that are available to be assigned to a logical volume.
Logical Volumes are a logical block device that is made up of portions of the Volume Group they fall under.
To create a logical volume, you must first assign a block device as a physical volume, then assign that physical volume to a volume group, and then create a logical volume from the extents of the volume group. Logical volumes are concatenated (additional device extents are added after the end of previous drive extents) by default, but can also be striped or mirrored just like raid as well. With concatenated volumes, there is much less risk of losing the entire volume's worth of data on a single device failure. You can attempt to remove the missing extents from the VG, and then use dd to byte-for-byte copy the remaining data to some temporary space and recover any complete plot files.
These examples assume a drive naming convention of /mnt/plot and that no loops are currently mounted. If you have snaps installed on ubuntu, they will be using the first available loop devices. In this example we will create an LV from the previous example of two 10TB drives' leftover space:
$ fallocate -l 90G /mnt/plot1/plot1.img
$ fallocate -l 90G /mnt/plot2/plot2.img
# losetup /dev/loop1 /mnt/plot1/plot1.img
# losetup /dev/loop2 /mnt/plot2/plot2.img
# pvcreate /dev/loop1 /dev/loop2
# vgcreate vgname /dev/loop1 /dev/loop2
# lvcreate -n lvname -l 100%FREE vgname
# mkfs.ext4 -m 0 -O ^has_journal -T largefile4 /dev/vgname/lvname
# mkdir /mnt/lvplots
# mount /dev/vgname/lvname /mnt/lvplots
In this example we will add a third 10TB drive's leftover space to the previously created Logical Volume:
$ fallocate -l 90G /mnt/plot3/plot3.img
# losetup /dev/loop3 /mnt/plot3/plot3.img
# pvcreate /dev/loop3
# vgextend vgname /dev/loop3
# umount /mnt/lvplots
# lvextend -r -l +100%FREE vgname/lvname
# mount /dev/vgname/lvname /mnt/lvplots
This is a basic script that can be run at startup as root after all drives have been mounted:
#!/usr/bin/bash
losetup /dev/loop1 /mnt/plot1/plot1.img
losetup /dev/loop2 /mnt/plot2/plot2.img
losetup /dev/loop3 /mnt/plot3/plot3.img
# Sometimes it takes a moment for the volume group to recognize itself after initial loop re-creation,
# this command should ensure that it is running before the mount command.
vgchange -a y vgname
mount /dev/vgname/lvname /mnt/lvplots
You can have the script run at startup using systemd, just make sure it runs after the mount events.
To prevent the loops from stalling system shutdown, it's also a good idea to run a script to disable them prior to shutting down:
#!/usr/bin/bash
umount /mnt/lvplots
losetup -d /dev/loop1
losetup -d /dev/loop2
losetup -d /dev/loop3
Using 100%FREE when creating the logical volume is fine if you intend to never adjust your hardware setup, but can be slightly inconvenient if you have to remove a drive. Instead you can leave a portion of the extents free so it is simpler to rearrange the Volume Group without having to shrink the Logical Volume and filesystem. As an alternative to this, you can use a spare drive as a temporary PV in the Volume Group just to hold the extents from the drive you are moving until you replace it.
A good practice might be to only assign extents in amounts equal to k32 plots. For example, if we have our 3 10TB drives from the previous examples, we would have 270GiB available to our logical volume, but this is not enough to fit 3x k32 plots so another ~66GiB would go to waste. Instead we could choose to only assign around 206GiB to the logical volume during creation like so: lvcreate -n lvname -L 206G vgname. This still leaves 66GiB unassigned, but will be easier to adjust extents later. If we were to then add another 10TB drive with another 90GiB to our VG, there would then be 360GiB available, so enough for an additional plot: lvextend -r -L +103G vgname/lvname. This still leaves 51GiB unassigned, but in the event we had to remove one of those drives, now we just need a temporary location for 40GiB of extents instead of a full 90GiB, and we shouldn't need to shrink the filesystem (which is always a risky operation).
If you would like to always have the ability to remove any drive at any time, you can create a temporary loop device on your root drive that matches the largest loop device from any of your drives. Then when you desire to remove the drive, simply add that loop device as a PV, then add it to the VG of your plots. Unmount the logical volume and then you need to move the physical extents (PE) off of the loop device of the drive you wish to remove, then you can remove the PV from the VG, remove the PV from the PV list, and finally, destroy the loop device. Here's an example of the full process if we wanted to remove the second 10TB drive:
$ fallocate -l 90G /home/username/tempPV.img
# losetup /dev/loop4 /home/username/tempPV.img
# pvcreate /dev/loop4
# vgextend vgname /dev/loop4
# umount /mnt/lvplots
# pvmove /dev/loop2
# vgreduce vgname /dev/loop2
# pvremove /dev/loop2
# losetup -d /dev/loop2