One problem about TDX CVM use host GPU by VFIO passthrough

[fffmonkeyking-new]

When I use qemu-kvm to start a tdx-cvm and hope the tdx-cvm can use host GPU by VFIO passthrough:
$ /usr/libexec/qemu-kvm
... ...
-object iommufd,id=iommufd0
-device pcie-root-port,id=pci.1,bus=pcie.0
-device vfio-pci,host=28:00.0,bus=pci.1,iommufd=iommufd0 -fw_cfg name=opt/ovmf/X-PciMmio64,string=262144
... ...

An error occurres:
qemu-kvm: -device vfio-pci,host=28:00.0,bus=pci.1,iommufd=iommufd0: vfio 0000:28:00.0: vfio /sys/bus/pci/devices/0000:28:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:28:00.0/vfio-dev/vfio0/dev"

I have bound my host gpu to vfio on the host:
$ lspci -nnk -s 28:00.0
28:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
Subsystem: LeadTek Research Inc. Device [107d:2685]
Kernel driver in use: vfio-pci
Kernel modules: nouveau

I can't seem to find the dev file in the /sys/bus/pci/devices/0000:28:00.0/vfio-dev/vfio0/ directory
$ ls -al /sys/bus/pci/devices/0000:28:00.0/vfio-dev/vfio0/
total 0
drwxr-xr-x 3 root root 0 Feb 21 14:23 .
drwxr-xr-x 3 root root 0 Feb 21 14:23 ..
lrwxrwxrwx 1 root root 0 Feb 21 15:10 device -> ../../../0000:28:00.0
drwxr-xr-x 2 root root 0 Feb 21 15:10 power
lrwxrwxrwx 1 root root 0 Feb 21 15:10 subsystem -> ../../../../../../../../../../class/vfio-dev
-rw-r--r-- 1 root root 4096 Feb 21 14:23 uevent

What exactly causes this error?

[Tan-YiFan]

RTX 4090 does not support confidential computing. You should use Hopper or Blackwell series GPU.

[fffmonkeyking-new]

RTX 4090 does not support confidential computing. You should use Hopper or Blackwell series GPU.

Yes @Tan-YiFan .

However, as described in this issue: NVIDIA/open-gpu-kernel-modules#787:
My use case: Let the LLM running in the TDX CVM use the A100/T100 GPU (due to cost reasons, we do not use the Hopper series GPU) for inference calculations.
Of course, I know and can accept the fact that the data is not safe after being copied from the CVM to the None CC GPU.

Also, Intel's https://github.com/intel/tdx-linux/tree/device-passthrough project provides a patch for TDX CVM to use the host GPU (cc or non-cc) via VFIO passthrough.

[fffmonkeyking-new]

In the first step, I will try to make TDX CVM successfully discover the none-cc-gpu device by VFIO passthrough.

Then I will see what the problem is with the GPU Driver for none-cc-gpu.
@Tan-YiFan

[Tan-YiFan]

Could you check whether the IOMMU is enabled on the CPU?

If enabled, dmesg | grep iommu_group would have a lot of output. Othervise, please add intel_iommu=on iommu=pt in grub command line, and then sudo update-grub and reboot the host.

[fffmonkeyking-new]

Could you check whether the IOMMU is enabled on the CPU?

If enabled, dmesg | grep iommu_group would have a lot of output. Othervise, please add intel_iommu=on iommu=pt in grub command line, and then sudo update-grub and reboot the host.

@Tan-YiFan 我已经在 /etc/default/grub 里添加了 intel_iommu=on iommu=pt

dmesg | grep iommu_group
没有任何输出

但是
dmesg | grep iommu | grep -i 28:00.0 有输出：
[ 2.298318] pci 0000:28:00.0: Adding to iommu group 18

[Tan-YiFan]

Does dmesg give more error about failed to load "/sys/bus/pci/devices/0000:28:00.0/vfio-dev/vfio0/dev"?
Additionally, could you give the output of ls /dev/vfio?

[fffmonkeyking-new]

Does dmesg give more error about failed to load "/sys/bus/pci/devices/0000:28:00.0/vfio-dev/vfio0/dev"?
报错：“qemu-kvm: -device vfio-pci,host=28:00.0,bus=pci.1,iommufd=iommufd0: vfio 0000:28:00.0: vfio /sys/bus/pci/devices/0000:28:00.0/vfio-dev: failed to load "/sys/bus/pci/devices/0000:28:00.0/vfio-dev/vfio0/dev"”
时 dmesg 的信息：
[ 2953.023711] cni0: port 4(veth93af7777) entered disabled state
[ 2953.052910] veth93af7777 (unregistering): left allmulticast mode
[ 2953.053413] veth93af7777 (unregistering): left promiscuous mode
[ 2953.053812] cni0: port 4(veth93af7777) entered disabled state
[ 2953.341556] cni0: port 4(veth694a0291) entered blocking state
[ 2953.341848] cni0: port 4(veth694a0291) entered disabled state
[ 2953.342184] veth694a0291: entered allmulticast mode
[ 2953.342509] veth694a0291: entered promiscuous mode
[ 2953.342829] cni0: port 4(veth694a0291) entered blocking state
[ 2953.343149] cni0: port 4(veth694a0291) entered forwarding state
[ 2953.409899] eth0: Caught tx_queue_len zero misconfig

Additionally, could you give the output of ls /dev/vfio?
ls -al /dev/vfio
total 0
drwxr-xr-x 2 root root 220 Feb 24 14:44 .
drwxr-xr-x 23 root root 3660 Feb 24 14:36 ..
crw------- 1 root root 509, 0 Feb 24 14:44 18
crw------- 1 root root 509, 1 Feb 24 14:44 19
crw------- 1 root root 509, 2 Feb 24 14:44 22
crw------- 1 root root 509, 3 Feb 24 14:44 27
crw------- 1 root root 509, 4 Feb 24 14:44 58
crw------- 1 root root 509, 5 Feb 24 14:44 59
crw------- 1 root root 509, 6 Feb 24 14:44 62
crw------- 1 root root 509, 7 Feb 24 14:44 67
crw-rw-rw- 1 root root 10, 196 Feb 24 14:44 vfio

[Tan-YiFan]

Is the command executed with sudo? hhh

I am not sure what's going wrong. Could you try booting a non-TDX VM with that GPU?

[fffmonkeyking-new]

Is the command executed with sudo? hhh
： yes

I am not sure what's going wrong. Could you try booting a non-TDX VM with that GPU?
: another error
qemu-kvm: -device vfio-pci,host=28:00.0: vfio 0000:28:00.0: group 18 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

my gpu devices:
$ lspci -nn | grep -i nvidia
28:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
28:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
29:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
29:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
2c:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
2c:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
31:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
31:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
a8:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
a8:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
a9:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
a9:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
ac:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
ac:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)
b1:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
b1:00.1 Audio device [0403]: NVIDIA Corporation AD102 High Definition Audio Controller [10de:22ba] (rev a1)

$ lspci -nnk -s 28:00.0
28:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
Subsystem: LeadTek Research Inc. Device [107d:2685]
Kernel driver in use: vfio-pci
Kernel modules: nouveau

$ lspci -nnk -s 29:00.0
29:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)
Subsystem: LeadTek Research Inc. Device [107d:2685]
Kernel driver in use: vfio-pci
Kernel modules: nouveau

$ find /sys/kernel/iommu_groups/ -type l | grep -E "28:00.0|29:00.0|2c:00.0"
/sys/kernel/iommu_groups/18/devices/0000:28:00.0
/sys/kernel/iommu_groups/22/devices/0000:2c:00.0
/sys/kernel/iommu_groups/19/devices/0000:29:00.0

28:00.0、29:00.0、2c:00.0、... 必须是在一个 iommu_groups 组里？

[Tan-YiFan]

Maybe 28:00.1 should be unbound from host?

[fffmonkeyking-new]

Maybe 28:00.1 should be unbound from host?
：yes

@Tan-YiFan

$ modprobe -r snd_hda_intel

这是成功了么？ 在non-TDX VM with that GPU：
$ lspci | grep -i nvidia
00:04.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090 D] (rev a1)

但是 for TDX VM, 仍然是相同的错误，可能 kvm / qemu / ovmf 没有正确的 patch 上 intel tdx-linux 的 patch.

[Tan-YiFan]

Yes, the GPU is able to be passed through to a non-TDX VM.

It seems that the error in CVM is at https://elixir.bootlin.com/qemu/v9.2.1/source/hw/vfio/iommufd.c#L210. Since I don't know what's happening, I would hack into this function to see what's different in non-TDX VM and TDX VM.

[fffmonkeyking-new]

@Tan-YiFan

For TDX-CVM, if I replace the
"
-object iommufd,id=iommufd0
-device pcie-root-port,id=pci.1,bus=pcie.0
-device vfio-pci,host=28:00.0,bus=pci.1,iommufd=iommufd0 -fw_cfg name=opt/ovmf/X-PciMmio64,string=262144
"
in the qemu startup command with
"
-device vfio-pci,host=28:00.0
",
the TDX-CVM can be started and the Nvidia-GPU-Device can be seen in the TDX-CVM:
$ lspci -nn | grep -i nvidia
00:04.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090 D] [10de:2685] (rev a1)

So, is it a problem with the IOMMU program or configuration?

[Tan-YiFan]

Maybe it's compatibility issue: https://www.qemu.org/docs/master/devel/vfio-iommufd.html#step-2-configure-qemu