Unable to launch the Guest OS in device-passthrough branch

[chenjikai111]

I want to pass a NVMe device to VM using the device-passthrough branch, but I ran into an error that the system couldn't start, it's stuck here:

# qemu-system-x86_64 -enable-kvm -drive file=/root/tdx-noble-24.04/guest-tools/image/tdx-guest-ubuntu-24.04-generic.qcow2,if=virtio -smp cores=20,threads=2,sockets=2 -m 5G -cpu host -netdev user,id=n1,ipv6=off,hostfwd=tcp::2022-:22 -device virtio-net-pci,netdev=n1,mac=7c:c2:55:9f:64:58 -nographic -object tdx-guest,id=tdx -object memory-backend-ram,id=mem0,size=5G -machine q35,kernel-irqchip=split,confidential-guest-support=tdx,memory-backend=mem0 -bios /root/git/edk2_sn/OVMF.fd -vga none -nodefaults -serial stdio

BdsDxe: loading Boot0007 "Ubuntu" from HD(15,GPT,73CBA394-9E5D-4AAC-9A82-73A0939BDFF9,0x2800,0x35000)/\EFI\ubuntu\shimx64.efi
BdsDxe: starting Boot0007 "Ubuntu" from HD(15,GPT,73CBA394-9E5D-4AAC-9A82-73A0939BDFF9,0x2800,0x35000)/\EFI\ubuntu\shimx64.efi
Loading Linux 6.8.0-41-generic ...
Loading initial ramdisk ...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Measured initrd data into PCR 9

I've patched QEMU and KVM, and my system has been activated TDX：
[ 3.787546] virt/tdx: module initialized

I didn't use the https://github.com/tianocore/edk2.git code to generate OVMF because there was a problem upgrading the subunits, and I replaced it with https://github.com/tlendacky/edk2.git > snp-fixes.

What is the cause of this mistake?

[dujh]

The current upstream OVMF does not support it.

[chenjikai111]

The current upstream OVMF does not support it.

You mean there's something wrong with the code in this branch https://github.com/tlendacky/edk2.git, right?
Should the operation of the steps based on wiki be no problem?

[dujh]

Neither one supports NVMe.

[chenjikai111]

Neither one supports NVMe.

Okay, thanks:）

[chenjikai111]

Neither one supports NVMe.

I would like to know if you are working on support for NVMe passthrough or not, however storage devices are a very important part in VM.

[dujh]

We don’t have plans for introducing NVMe support at this time.

[chenjikai111]

@dujh Hello, I tried to start TD with the simplest parameters:

qemu-system-x86_64 --enable-kvm -m 2G -smp 16 -name td,process=td,debug-threads=on -cpu host -object tdx-guest,id=tdx -machine q35,kernel_irqchip=split,confidential-guest-support=tdx,hpet=off -bios /usr/share/ovmf/OVMF.fd -drive file=./tdx-guest-ubuntu-24.04-generic.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0

And I got the following error :

2025-05-15T00:08:39.497451+08:00 sriov kernel: BUG: kernel NULL pointer dereference, address: 0000000000000090
2025-05-15T00:08:39.497476+08:00 sriov kernel: #PF: supervisor read access in kernel mode
2025-05-15T00:08:39.497477+08:00 sriov kernel: #PF: error_code(0x0000) - not-present page
2025-05-15T00:08:39.497479+08:00 sriov kernel: PGD 12c402067 P4D 0
2025-05-15T00:08:39.497481+08:00 sriov kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
2025-05-15T00:08:39.497482+08:00 sriov kernel: CPU: 96 PID: 5438 Comm: CPU 15/KVM Not tainted 6.9.0-rc7+ #3
2025-05-15T00:08:39.497483+08:00 sriov kernel: Hardware name: Dell Inc. PowerEdge R760/09XV41, BIOS 2.1.5 03/14/2024
2025-05-15T00:08:39.497484+08:00 sriov kernel: RIP: 0010:vmx_update_cpu_dirty_logging+0x46/0x1c0
2025-05-15T00:08:39.497485+08:00 sriov kernel: Code: 01 0f 87 72 f8 17 01 41 83 e4 01 0f 84 09 01 00 00 f6 83 f0 01 00 00 01 75 6d 48 8b 03 8b 88 38 11 00 00 48 8b 83 20 1c 00 00 <8b> 90 90 00 00 00 85 c9 74 3e 41 89 d4 41 81 cc 00 00 02 00 41 39
2025-05-15T00:08:39.497487+08:00 sriov kernel: RSP: 0018:ff76d5c265f97ca8 EFLAGS: 00010246
2025-05-15T00:08:39.497488+08:00 sriov kernel: RAX: 0000000000000000 RBX: ff4e802edc980000 RCX: 0000000000000001
2025-05-15T00:08:39.497489+08:00 sriov kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff4e802edc980000
2025-05-15T00:08:39.497491+08:00 sriov kernel: RBP: ff76d5c265f97cc0 R08: 0000000000000000 R09: 0000000000000000
2025-05-15T00:08:39.497492+08:00 sriov kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
2025-05-15T00:08:39.497494+08:00 sriov kernel: R13: 0000000000000000 R14: ff4e802ecc0ab000 R15: ff4e802edc980038
2025-05-15T00:08:39.497496+08:00 sriov kernel: FS: 0000780dd74006c0(0000) GS:ff4e806cbfa00000(0000) knlGS:0000000000000000
2025-05-15T00:08:39.497497+08:00 sriov kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2025-05-15T00:08:39.497499+08:00 sriov kernel: CR2: 0000000000000090 CR3: 000000010b28c001 CR4: 0000000000f73ef0
2025-05-15T00:08:39.497499+08:00 sriov kernel: PKRU: 55555554
2025-05-15T00:08:39.497501+08:00 sriov kernel: Call Trace:
2025-05-15T00:08:39.497502+08:00 sriov kernel:
2025-05-15T00:08:39.497504+08:00 sriov kernel: ? show_regs+0x6c/0x80
2025-05-15T00:08:39.497506+08:00 sriov kernel: ? __die+0x24/0x80
2025-05-15T00:08:39.497507+08:00 sriov kernel: ? page_fault_oops+0x175/0x5a0
2025-05-15T00:08:39.497508+08:00 sriov kernel: ? kvm_arch_vcpu_load+0x7c/0x240
2025-05-15T00:08:39.497510+08:00 sriov kernel: ? do_user_addr_fault+0x306/0x6f0
2025-05-15T00:08:39.497511+08:00 sriov kernel: ? finish_task_switch.isra.0+0x10a/0x340
2025-05-15T00:08:39.497512+08:00 sriov kernel: ? exc_page_fault+0x82/0x1b0
2025-05-15T00:08:39.497513+08:00 sriov kernel: ? asm_exc_page_fault+0x27/0x30
2025-05-15T00:08:39.497513+08:00 sriov kernel: ? vmx_update_cpu_dirty_logging+0x46/0x1c0
2025-05-15T00:08:39.497514+08:00 sriov kernel: kvm_arch_vcpu_ioctl_run+0x16d0/0x1720
2025-05-15T00:08:39.497515+08:00 sriov kernel: ? syscall_exit_to_user_mode+0x81/0x270
2025-05-15T00:08:39.497516+08:00 sriov kernel: kvm_vcpu_ioctl+0x257/0x9c0
2025-05-15T00:08:39.497516+08:00 sriov kernel: ? do_futex+0x18e/0x260
2025-05-15T00:08:39.497517+08:00 sriov kernel: ? __x64_sys_futex+0x12a/0x200
2025-05-15T00:08:39.497518+08:00 sriov kernel: ? do_syscall_64+0x8b/0x180
2025-05-15T00:08:39.497518+08:00 sriov kernel: __x64_sys_ioctl+0xa0/0xf0
2025-05-15T00:08:39.497520+08:00 sriov kernel: x64_sys_call+0x143b/0x25c0
2025-05-15T00:08:39.497521+08:00 sriov kernel: do_syscall_64+0x7e/0x180
2025-05-15T00:08:39.497523+08:00 sriov kernel: ? syscall_exit_to_user_mode+0x81/0x270
2025-05-15T00:08:39.497524+08:00 sriov kernel: ? do_syscall_64+0x8b/0x180
2025-05-15T00:08:39.497525+08:00 sriov kernel: ? irqentry_exit_to_user_mode+0x76/0x270
2025-05-15T00:08:39.497526+08:00 sriov kernel: ? irqentry_exit+0x43/0x50
2025-05-15T00:08:39.497527+08:00 sriov kernel: ? exc_page_fault+0x93/0x1b0
2025-05-15T00:08:39.497527+08:00 sriov kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
2025-05-15T00:08:39.497528+08:00 sriov kernel: RIP: 0033:0x780e75924ded
2025-05-15T00:08:39.497530+08:00 sriov kernel: Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
2025-05-15T00:08:39.497532+08:00 sriov kernel: RSP: 002b:0000780dd73ff5a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
2025-05-15T00:08:39.497532+08:00 sriov kernel: RAX: ffffffffffffffda RBX: 00005907a6ddbe50 RCX: 0000780e75924ded
2025-05-15T00:08:39.497533+08:00 sriov kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000002d
2025-05-15T00:08:39.497533+08:00 sriov kernel: RBP: 0000780dd73ff5f0 R08: 0000000000000000 R09: 0000000000000000
2025-05-15T00:08:39.497534+08:00 sriov kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000780dd74006c0
2025-05-15T00:08:39.497536+08:00 sriov kernel: R13: fffffffffffffb08 R14: 0000000000000007 R15: 00007ffefa1595a0
2025-05-15T00:08:39.497537+08:00 sriov kernel:
2025-05-15T00:08:39.497538+08:00 sriov kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq xt_CHECKSUM snd_seq_device snd_timer snd xt_MASQUERADE xt_conntrack soundcore ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables bridge vsock_loopback vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb stp vsock llc overlay qrtr binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 iaa_crypto sha1_ssse3 aesni_intel crypto_simd cryptd cmdlinepart spi_nor rapl pmt_telemetry qat_4xxx mtd pmt_class intel_sdsi dell_smbios intel_qat ipmi_ssif intel_cstate dcdbas dax_hmem idxd dell_wmi_descriptor wmi_bmof mgag200 mei_me isst_if_mmio spi_intel_pci i2c_i801 cxl_acpi isst_if_mbox_pci crc8 mei spi_intel idxd_bus i2c_smbus
2025-05-15T00:08:39.497543+08:00 sriov kernel: intel_vsec isst_if_common authenc i2c_algo_bit cxl_core i2c_ismt acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler einj acpi_power_meter input_leds mac_hid sch_fq_codel msr parport_pc ppdev lp parport nvme_fabrics efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 hid_generic usbhid hid nvme nvme_core megaraid_sas crc32_pclmul tg3 ahci xhci_pci libahci nvme_auth xhci_pci_renesas wmi pinctrl_emmitsburg
2025-05-15T00:08:39.497544+08:00 sriov kernel: CR2: 0000000000000090
2025-05-15T00:08:39.497545+08:00 sriov kernel: ---[ end trace 0000000000000000 ]---

Do you know the reason for this error?

[dujh]

It looks like it is configuration problem. Could you make sure you folllow the instruction?

[chenjikai111]

I installed it according to this guide:https://docs.nvidia.com/cc-deployment-guide-tdx.pdf,
and I just upgraded the TDX Module version.

[Qiangcy]

The calltrace shows it runs into the problem with RIP: 0010:vmx_update_cpu_dirty_logging+0x46/0x1c0. It is weird because it would just return if it is a TD vcpu.

[chenjikai111]

It's really weird. I made sure TDX was on :
[ 2.288798 ] Vit / tdx : BIOS enabled : private KeyID range [ 32, 64)
I printed vcpu->arch.hflags = 0, and did not assign HF_GUEST_MASK.

[chenjikai111]

@dujh Hello, I have a confusion about the OVMF.
Because I find that support for PCIe devices, including NVMe devices and GPU, had been added to linux-6.8.X-intel, Could you tell me what support is needed for PCIe devices in OVMF?