Describes how to setup Proxmox VE 6.2 with ZFS and device passthrough (Intel CPU).
Supermicro X11SCZ-F using C246 chipset.
Intel I3-9300 with integrated UHD630 GPU.
Micron 18ASF2G72AZ-2G3B1, 16GiB * 4 sticks
Two NVMe SSD drives for ZFS on root mirror (SK hynix Gold P31 1TB, SHGP31-1000GM-2)
Enable VT-d, VT-x, etc.
Enable ASPM or set to Auto.
Enable UEFI boot.
Set primary display to PCI and enable Internal Graphics option to be able to passthrough integrated GPU.
Edit systemd-boot file nano /etc/kernel/cmdline by adding intel_iommu=on iommu=pt options.
Apply changes by executing pve-efiboot-tool refresh
Add required modules nano /etc/modules:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Apply changes by executing update-initramfs -u -k all
Reboot the host
Execute dmesg | grep -e DMAR -e IOMMU and check the output for:
[ 0.103246] DMAR: IOMMU enabled
[ 0.172321] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.175358] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.913016] DMAR: Intel(R) Virtualization Technology for Directed I/O
Prevent passthrough devices from host loading by editing nano /etc/modprobe.d/blacklist.conf. For example:
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
blacklist i915
All functions [Y]
ROM-Bar [Y]
PCI-Express [Y]
Tested on Intel I350-T4. Each nic must be in its own iommu group. When passing through individual nics, do not enable All Functions or ROM-Bar.
rm /etc/apt/sources.list.d/pve-enterprise.list
echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
apt update
apt full-upgrade
Proxmox writes lots of logs to disk and will wear out SSDs fast. Remove any logging you don't need. For standalone PVE, some services can be disabled.
systemctl stop pve-ha-lrm.service
systemctl stop pve-ha-crm.service
systemctl disable pve-ha-lrm.service
systemctl disable pve-ha-crm.service
For single box PVE, make timer not run too often by setting OnCalendar=monthly
systemctl edit --full pvesr.timer
systemctl daemon-reload
Enable SSD autotrim.
zpool set autotrim=on rpool
To manually trim, execute zpool trim.
Check time before async writes are written to disk. Increased timeout reduces SSD wear.
cat /sys/module/zfs/parameters/zfs_txg_timeout
Set async write timeout temporarily to see impact.
echo 30 > /sys/module/zfs/parameters/zfs_txg_timeout
To preserve zfs_txg_timeout across reboots, modify config to include options zfs zfs_txg_timeout=30 line.
nano /etc/modprobe.d/zfs.conf
update-initramfs -u
pve-efiboot-tool refresh
Qemu/KVM Virtual Machines Wiki
Qemu/KVM Virtual Machines Docs