virtio-net control queue returns VIRTIO_NET_ERR for unsupported commands, breaking Windows NetKVM driver

[CMGS]

Describe the bug

Cloud Hypervisor's virtio-net control virtqueue returns VIRTIO_NET_ERR for all command classes except VIRTIO_NET_CTRL_MQ and VIRTIO_NET_CTRL_GUEST_OFFLOADS. This causes the Windows VirtIO NetKVM driver (virtio-win >= 0.1.271) to fail during initialization with NDIS Event 10317: "The miniport has detected an internal error" and PnP Problem Code 43 (CM_PROB_FAILED_POST_START).

The unsupported command classes that trigger the failure include VIRTIO_NET_CTRL_RX (class 0), VIRTIO_NET_CTRL_MAC (class 1), VIRTIO_NET_CTRL_VLAN (class 2), and VIRTIO_NET_CTRL_ANNOUNCE (class 3). QEMU handles all of these correctly, so the same drivers work there.

Only virtio-win 0.1.240 (July 2023) works, as it silently tolerates the error. 0.1.271+ results in a non-functional NIC, and 0.1.285+ immediately tears down the miniport via NdisMRemoveMiniport() (commit 50e7db9 in virtio-win -- "indicate driver error on unexpected CX behavior").

A safe fix would be to return VIRTIO_NET_OK for unimplemented command classes instead of VIRTIO_NET_ERR, as these commands (RX filtering, MAC table, VLAN, gratuitous ARP) can be safely no-op'd per the virtio spec.

To Reproduce

1. Install Windows 11 25H2 with virtio-win >= 0.1.271 NetKVM driver (installed via QEMU with disable-legacy=on)
2. Boot the image with Cloud Hypervisor (see VM configuration below)
3. In SAC CMD or RDP, run pnputil /enum-devices /class Net -- VirtIO adapter shows Status: Problem, Problem Code: 43
4. Event Viewer -> System log shows NDIS Event ID 10317: "Miniport Red Hat VirtIO Ethernet Adapter had event Fatal error: The miniport has detected an internal error"
5. ipconfig shows no network adapter / no IP address
6. Repeat with virtio-win 0.1.240 -- networking works normally

Version

Output of cloud-hypervisor --version:

cloud-hypervisor v50.2

Also reproduced on v51.1. Pre-built binaries from GitHub releases.

VM configuration

What command line did you run:

cloud-hypervisor \
  --kernel CLOUDHV.fd \
  --disk path=windows-11-25h2.qcow2,num_queues=2,queue_size=512 \
  --cpus boot=2,kvm_hyperv=on \
  --memory size=4G \
  --net tap=vmtap0,mac=52:54:00:12:34:56,num_queues=2 \
  --serial tty \
  --console off

Guest OS version details: Windows 11 25H2 (build 26100.8037)

Host OS version details: Ubuntu, Linux 6.17.0-1009-gcp, KVM

Drivers tested:

• virtio-win 0.1.240 -- works
• virtio-win 0.1.271 -- broken
• virtio-win 0.1.285 -- broken
• attestation-virtio-win-prewhql 0.1.298 -- broken

Logs

Cloud Hypervisor log (-v) shows no error -- the unsupported control command is only logged as a warning:

cloud-hypervisor: WARN: net_util/src/ctrl_queue.rs -- Unsupported command class

Windows guest pnputil output (broken with 0.1.271+):

Instance ID:    PCI\VEN_1AF4&DEV_1041&SUBSYS_10411AF4&REV_01\3&11583659&0&10
Device Description:     Red Hat VirtIO Ethernet Adapter #2
Status:                 Problem
Problem Code:           43 (0x2B) [CM_PROB_FAILED_POST_START]
Driver Name:            oem0.inf

Windows Event Log (NDIS):

Source: Microsoft-Windows-NDIS
Event ID: 10317
Level: Error
Description: Miniport Red Hat VirtIO Ethernet Adapter #2, {dc6ea326-...},
  had event Fatal error: The miniport has detected an internal error

[liuw]

A safe fix would be to return VIRTIO_NET_OK for unimplemented command classes instead of VIRTIO_NET_ERR, as these commands (RX filtering, MAC table, VLAN, gratuitous ARP) can be safely no-op'd per the virtio spec.

In https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html 5.1.6.5.2.1 Device Requirements: Setting MAC Address Filtering.

The device SHOULD drop incoming packets which have a destination MAC which matches neither the mac (or that set with VIRTIO_NET_CTRL_MAC_ADDR_SET) nor the MAC filtering table.

Did you see mac filtering command class coming through? I think we may want to be a bit stricter here?

On the other hand, I asked an LLM to fix this issue. It came to the conclusion as you did. Here is the code. liuw@b2f53c7. I reviewed and modified the code a bit, but have not tested it. I need to find some copious cycles to check if we need to drain the descriptor chains.

[CMGS]

@liuw ah good point, let me verify MAC filtering later, will update here asap.

[CMGS]

@liuw tested your commit on top of current main with a Windows 11 25H2 guest using virtio-win 0.1.285. the VM crashes before the NetKVM driver reaches the control queue stage, so I was unable to observe whether MAC filtering commands come through.

The root cause is a BAR reprogramming failure. The 0.1.285 NetKVM driver attempts to move the virtio-net device BAR to a high address (>4GB), which the CH MMIO allocator rejects:

ERROR:pci/src/bus.rs:408 -- Failed moving device BAR: failed allocating new MMIO range: 0x3fff3ff00000->0x3fff3ff80000(0x80000)
ERROR:virtio-devices/src/device.rs:247 -- Number of enabled queues lower than min: 1 vs 2                                                            
Fatal error: VmmThread(ActivateVirtioDevices(...))   

Windows remapping virtio BARs above 4GB. With 0.1.240 the driver doesn't do this remapping so everything works, but 0.1.285 triggers it and CH exits.

[CMGS]

@liuw i cherry-picked your commit and depends on it made a small change, then the latest virtio-net works. can check https://github.com/CMGS/cloud-hypervisor/tree/fix/virtio-net-ctrl-queue here

[liuw]

@liuw i cherry-picked your commit and depends on it made a small change, then the latest virtio-net works. can check https://github.com/CMGS/cloud-hypervisor/tree/fix/virtio-net-ctrl-queue here

Thanks. Did you see the mac filtering command? The commit I gave you put it in the tolerated list, but I would rather remove it if possible.

[CMGS]

@liuw

     cloud-hypervisor:   0.016586s: <vmm> WARN:net_util/src/open_tap.rs:84 -- Tap vmtap0 already exists. IP configuration will not be overwritten.
     cloud-hypervisor:  17.419643s: <vcpu1> INFO:virtio-devices/src/transport/pci_device.rs:1250 -- _virtio-pci-_net1: Needs activation; writing to activate event fd
     cloud-hypervisor:  17.419693s: <vcpu1> INFO:virtio-devices/src/transport/pci_device.rs:1255 -- _virtio-pci-_net1: Needs activation; returning barrier
     cloud-hypervisor:  17.420010s: <vmm> INFO:event_monitor/src/lib.rs:113 -- Event: source = virtio-device event = activated id = _net1
     cloud-hypervisor:  17.420025s: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:325 -- _virtio-pci-_net1: Waiting for barrier
     cloud-hypervisor:  17.420038s: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:327 -- _virtio-pci-_net1: Barrier released
     cloud-hypervisor:  17.420126s: <_net1_ctrl> INFO:net_util/src/ctrl_queue.rs:146 -- Reprogramming tap offload with features: 386
     cloud-hypervisor:  17.420179s: <_net1_qp0> INFO:virtio-devices/src/net.rs:339 -- Listener registered at start
     cloud-hypervisor:  17.427885s: <_net1_ctrl> INFO:net_util/src/ctrl_queue.rs:146 -- Reprogramming tap offload with features: 386
     cloud-hypervisor: 275.465454s: <_net1_ctrl> INFO:virtio-devices/src/epoll_helper.rs:218 -- KILL_EVENT received, stopping epoll loop
     cloud-hypervisor: 275.465454s: <_net1_qp0> INFO:virtio-devices/src/epoll_helper.rs:218 -- KILL_EVENT received, stopping epoll loop

no tolerated command ( mac filtering )

[liuw]

no tolerated command ( mac filtering )

Okay. Thanks for testing. I will drop mac filtering from the tolerated list.