Use of qemu PCIe to PCI bridges on x86 causes multiple problems

[dgibson]

Description of problem

On x86 systems, the Kata qemu backend always adds a PCI bridge. This is qemu's pci-bridge device type, which will be a PCI to PCI bridge on the pc machine type and PCIe to PCI on q35. Hotplugged block, net and vhost-user devices always go on this bridge. Hotplugged VFIO devices also go on the bridge by default (if hotplug_vfio_on_root_bus=true is not set).

Using a bridge like this conveniently provides 32 pluggable slots and makes pc and q35 behave more similarly. However, at least with current qemu and guest kernel it forces use of the SHPC hotplug protocol, which has some severe drawbacks.

SHPC is designed with physical devices and human operators in mind, and so has a 5s delay built into the protocol to allow accidental plugs to be reversed. Since a 5s delay in startup isn't acceptable, we work around this in the agent by forcing a PCI rescan which bypasses the proper operation of SHPC and locates the device early. Unfortunately, that workaround causes other problems

• An SHPC hotplug can sometimes race with the rescan in a way which means that the guest kernel misinterprets the SHPC interrupt as a request to unplug the device, meaning the device appears very briefly then is removed again
• For passed through VFIO devices, an even more severe error can occur which can put the device into an unusable state until a host reboot.

For these reasons #683 exists to remove the rescan from the agent, however doing so means SHPC hotplugs will have an unacceptable delay.

For VFIO devices, the use of a PCI bridge has additional problems beyond SHPC.

• If the passed through device is PCI-Express (essentially all likely use cases), then putting it under the bridge means the guest will see it as a plain PCI device instead. Depending on the device this may mean that the guest can't drive it properly.
• Because pre-Express PCI bridges don't preserve Requestor IDs, all devices behind a PCI bridge will always be in the same IOMMU group in the guest, even if they are in separate groups in the host. That means
  • If the container was intending to use separate VFIO devices from separate userspace drivers (e.g. DPDK), they won't be able to
  • Worse, if the container has both attached block and net devices (managed by the guest kernel) and vfio devices they intended to use with userspace drivers, they won't be able to at all (a whole IOMMU group must belong either to the kernel or to userspace).

[dgibson]

@marcel-apf @fidencio for your attention.

[marcel-apf]

As discussed offline with @dgibson, since the ACPI hoptlug for the PCIe-PCI bridge solves the problem only partially, it seems the best approach would be to use a single slot with 8 PCIe root ports (functions) and use them for hot-plug.
Two issues with this approach:

• What if an user wants more devices? It can be configurable, I assume that 8 ports would be OK for most use cases.
• More root ports increases the boot time: We can try to tackle the issue and speed up the discovery, it will be faster than the SHPC's 5 seconds anyway.

[dgibson]

Right. There are some other things that will have to be sorted out to follow that approach as well:

• Currently, Kata supports (in fact, defaults to) the pc machine type, which can't take root ports, so we'll need to sort out what to do with that (Kata qemu backend handles machine types poorly #1727 is somewhat related)
• Neither the runtime nor the agent currently handle non-zero functions properly. That at least isn't too hard to fix now that I've implemented the PCI path stuff

[fidencio]

Right. There are some other things that will have to be sorted out to follow that approach as well:

* Currently, Kata supports (in fact, defaults to) the `pc` machine type, which can't take root ports, so we'll need to sort out what to do with that (#1727 is somewhat related)

@dgibson, @marcel-apf,
I don't think we should spend much time figuring out what to do with pc machine type, as I think we should rather stop using it in favour of q35.

/cc @devimc @kata-containers/architecture-committee for inputs on moving on and mostly caring about the q35 machine type.

[devimc]

@fidencio +1 for q35

@dgibson when I implemented this (hotplug devices, etc) in kata a couple of year ago, I wrote a PoC for using root ports,
in addition to the limitations already mentioned by @marcel-apf, memory footprint was also impacted negatively, but the main reason for not using them (root ports) was the limitation in the number of devices supported, we don't know how many devices will be attached to the POD, so I suggest (if this is possible), to continue using PCI(e) bridges for non-VFIO devices (i.e block devices, virtiofs, etc) and rootports for VFIO devices (GPUs, NICs, etc), since we don't know how many VFIO devices will be hot plugged into the POD, the default number of root ports should be 0 and the user should be able to change it through the configuration file and/or annotations, what do you think ?

[marcel-apf]

@fidencio another +1 for q35, I don't see a reason to continue to support multiple machines in the kata, while q35 has features the pc machine doesn't support.

@devimc regarding the issue of not knowing how many devices will be hotplugged, we have the same problem with the pc machine where we have about 16 (20?) empty slots. What if the user needs more? We will need to add another PCI-PCI bridge to have more available slots (at configuration time).

In q35 the problem is we have no empty slots by default, but what if we would? I would re-check the effects of adding a single device (having 8 PCI Root Ports as its PCI functions) - both memory footprint and the boot time. In this way q35 will behave somehow similar to the pc machine, but with less available slots.

About plugging the virtio devices into a PCI slot (Behind a PCIe-PCI bridge), AFAIK we will lose the "Pure Virtio 1.X" mode (disable-legacy=on,disable-modern=off) and we will support only Transitional virtio (disable-legacy=off,disable-modern=off), I can't say I can measure the impact, however the whole point of using q35 is to avoid the legacy mode, and that would do the opposite.

I like the configuration approach, but I do think we should have 8(less? see below) pcie-root-ports available by default and let the user add more, however I like way more the dynamic approach based on annotations, if feasible.

One last note about the concern of using the pcie-root-ports. We can actually plug more than one virtio NIC (or other device type) to a pcie-root-port. We can plug up to 8 virtio-pci-{} functions into a single pcie-root-port. If the virtio devices support ARI (do they?) we can have up to 256 virtio-pci-{} functions into that pcie-root-port. That works also for hotplug.

If we don't care about the hot-unplug granularity (the PCI functions hot-plugged together must be hot-unplugged together), something like this could work:

pcie-root-bus
|
| - slot - pcie-root-port - func 1 virtio-pci...
| ----------------------------- func 2 virtio-pci...
| ----------------------------- ...
| --------- pcie-root-port - empty [can be used to hotplug several vfio/virtio PCI functions]
| --------- ...

In conclusion 1 slot, 8 pcie-root-ports, up to 8 PCI functions per pcie-root-port can give us up to 64 PCI functions.

[devimc]

@marcel-apf

About plugging the virtio devices into a PCI slot (Behind a PCIe-PCI bridge), AFAIK we will lose the "Pure Virtio 1.X" mode (disable-legacy=on,disable-modern=off) and we will support only Transitional virtio (disable-legacy=off,disable-modern=off), I can't say I can measure the impact, however the whole point of using q35 is to avoid the legacy mode, and that would do the opposite.

it would be great to know/measure what's the impact of this (if any)

I like the configuration approach, but I do think we should have 8(less? see below) pcie-root-ports available by default and let the user add more, however I like way more the dynamic approach based on annotations, if feasible.

we'll be wasting resources (CPU/Memory) if these root ports are only used for VFIO devices, since not all workloads need VFIO

If we don't care about the hot-unplug granularity (the PCI functions hot-plugged together must be hot-unplugged together),

we do care, devices must be hot unplugged when the container is restarted or destroyed, so maybe we should have 1 root port per container?

---

I'm not against using root ports for hot plugging alll the devices, but If we are going to follow that path, we should at least know the impact of that decision and evaluate it (are we willing to pay for it?)

[marcel-apf]

@devimc

@marcel-apf

About plugging the virtio devices into a PCI slot (Behind a PCIe-PCI bridge), AFAIK we will lose the "Pure Virtio 1.X" mode (disable-legacy=on,disable-modern=off) and we will support only Transitional virtio (disable-legacy=off,disable-modern=off), I can't say I can measure the impact, however the whole point of using q35 is to avoid the legacy mode, and that would do the opposite.

it would be great to know/measure what's the impact of this (if any)

I suspect the impact would be related to some features that may or may not influence the performance. For example leaving disable-legacy=off (transitional mode) will allow the guest to use IO instead of MMIO to communicate with the PCI devices. For kata we control the guest, so I suppose we can check and "fix" if necessary. In some cases IO access can be actually faster, the point remains we need to be aware of it.

I totally agree we need to understand all the implications.

I like the configuration approach, but I do think we should have 8(less? see below) pcie-root-ports available by default and let the user add more, however I like way more the dynamic approach based on annotations, if feasible.

we'll be wasting resources (CPU/Memory) if these root ports are only used for VFIO devices, since not all workloads need VFIO

CPU will be used only on the system boot to configure the devices, after that we are ok. We do increase the memory footprint as both QEMU and the guest will allocate resources for them. It would be really interesting to see how much.
We also waste the IO/MMIO ranges allocated to the pcie-root-ports not being used. IO resources tend to be scarce, but "modern" virtio devices and PCIe devices don't need IO in order to work.

However my intention was to use the pcie-root-ports for all devices, not only VFIO.

If we don't care about the hot-unplug granularity (the PCI functions hot-plugged together must be hot-unplugged together),

we do care, devices must be hot unplugged when the container is restarted or destroyed, so maybe we should have 1 root port per container?

That could work! Sorry for the possibly dumb question, but do we know the pod configuration (containers count, sriov devices for each container, other pertinent information that can help us to compute the number of pcie-root-ports needed) by the time we start the kata vm? If so, we only need to know if we need only 1 pcie-root-port (8 PCI functions) or more for each container.

I'm not against using root ports for hot plugging alll the devices, but If we are going to follow that path, we should at least know the impact of that decision and evaluate it (are we willing to pay for it?)

Agreed.

[devimc]

@marcel-apf

That could work! Sorry for the possibly dumb question, but do we know the pod configuration (containers count, sriov devices for each container, other pertinent information that can help us to compute the number of pcie-root-ports needed) by the time we start the kata vm? If so, we only need to know if we need only 1 pcie-root-port (8 PCI functions) or more for each container.

we don't, containers are created one at time and the runtime (or kata-shim) only knows the information of the containers that is being created, sadly (I think) user or cluster operator should provide that magic number (root ports) when the sandbox (POD) is created

[marcel-apf]

@devimc

@marcel-apf

That could work! Sorry for the possibly dumb question, but do we know the pod configuration (containers count, sriov devices for each container, other pertinent information that can help us to compute the number of pcie-root-ports needed) by the time we start the kata vm? If so, we only need to know if we need only 1 pcie-root-port (8 PCI functions) or more for each container.

we don't, containers are created one at time and the runtime (or kata-shim) only knows the information of the containers that is being created,

too bad

sadly (I think) user or cluster operator should provide that magic number (root ports) when the sandbox (POD) is created

I think the pc machine "solves" the issue by having about 20 empty slots and plugging each PCI function into a different slot, seeing it as a "whole" PCI device. It will suffer the exact same problem if a pod will have too many containers/devices.

For q35 we can assume we will not need more than 1 pcie-root-port per container (and make it configurable), but how many containers should we assume (and make it also configurable)? ...

I am back to my previous "configuration", one slot, 8 pci-root-ports (assume max 8 containers), 8 PCI functions per container.
The cluster admin should be able to configure both values. I am not convinced yet a VFIO device will require a "whole" pcie-root-port.

All of the above pending the "cost" research.

[devimc]

@marcel-apf

how many containers should we assume (and make it also configurable)?

this is impossible

I am back to my previous "configuration", one slot, 8 pci-root-ports (assume max 8 containers), 8 PCI functions per container.
The cluster admin should be able to configure both values.

sounds good to me, but this new "limitation" should be discussed (may in the next arch meeting? cc @kata-containers/architecture-committee) and if this is accepted, very well documented.

I am not convinced yet a VFIO device will require a "whole" pcie-root-port.

what about Large Bar Space PCI devices (GPUs > 4GB)? such devices may need the whole root port

[marcel-apf]

@devimc

@marcel-apf

I am back to my previous "configuration", one slot, 8 pci-root-ports (assume max 8 containers), 8 PCI functions per container.
The cluster admin should be able to configure both values.

sounds good to me, but this new "limitation" should be discussed (may in the next arch meeting? cc @kata-containers/architecture-committee) and if this is accepted, very well documented.

Sure.

I am not convinced yet a VFIO device will require a "whole" pcie-root-port.

what about Large Bar Space PCI devices (GPUs > 4GB)? such devices may need the whole root port

In this case the Guest OS will re-trigger resources allocation no matter if there is one or multiple PCI functions hot-plugged into the pcie-root-port. The reason is the firmware will not reserve enough MMIO (for devices with huge BARs) for an empty pcie-root-port. Once the resources re-allocation is triggered, all the PCI functions under the pcie-root-port will be re-evaluated and MMIO space will be allocated.

BTW, not directly related, we do have a way to speed things up and give a hint to the firmware regarding how much MMIO (64 bit) should allocate to a pcie-root-port so the Guest OS will not need to re-allocate:
-device pcie-root-port,pref64-reserve=8G
The IOMMU space of 64 bit VMs is not that expensive and we can preallocate "enough" MMIO to not trigger the Guest OS MMIO re-allocation, speeding things up.

[dgibson]

@dgibson, @marcel-apf,
I don't think we should spend much time figuring out what to do with pc machine type, as I think we should rather stop using it in favour of q35.

I was pretty much hoping that would be the opinion.

[dgibson]

@dgibson when I implemented this (hotplug devices, etc) in kata a couple of year ago, I wrote a PoC for using root ports,
in addition to the limitations already mentioned by @marcel-apf, memory footprint was also impacted negatively, but the main reason for not using them (root ports) was the limitation in the number of devices supported, we don't know how many devices will be attached to the POD, so I suggest (if this is possible), to continue using PCI(e) bridges for non-VFIO devices (i.e block devices, virtiofs, etc) and rootports for VFIO devices (GPUs, NICs, etc), since we don't know how many VFIO devices will be hot plugged into the POD, the default number of root ports should be 0 and the user should be able to change it through the configuration file and/or annotations, what do you think ?

I'm afraid that won't do what we need. The non-VFIO devices will still be using SHPC, which means using the rescan hack to cut down the startup time. The rescan is global (to a whole PCI domain), which means it can mess up VFIO devices even if they're on root ports.

[dgibson]

@marcel-apf using multiple root ports in a single slot via multifunction sounds good to me (though we will need Kata changes to accomodate it). I'm much more hesitant about placing different devices we want to attach as different functions of the same virtual PCI slot. For one thing, I thought qemu's hotplug granularity was effectively whole slot, not just "plugged as a group must be unplugged as a group".

I'm even more hesitant about rearranging functions for VFIO devices. Here we're passing through a real PCI device that would be typically expected to occupy a slot, so I'd be concerned that userspace drivers that aren't expecting it to be a function in a multifunction slot will get confused.

[marcel-apf]

@dgibson

@marcel-apf using multiple root ports in a single slot via multifunction sounds good to me (though we will need Kata changes to accomodate it). I'm much more hesitant about placing different devices we want to attach as different functions of the same virtual PCI slot. For one thing, I thought QEMU's hotplug granularity was effectively whole slot, not just "plugged as a group must be unplugged as a group".

Indeed, there is room for only one PCI device per slot, but a PCI device may have several PCI functions and QEMU fully supports hot-plug of multi-function PCI devices. The hotplug operations are per slot/device and this is the reason we need to hotplug/hot-unplug all the PCI functions as a single operation.

I see no reason why we shouldn't hotplug at least the virtio-pci-{type} as functions of the same device. Doing so will actually save PCI bus numbers, since each PCIe device uses a "whole" PCIe bus and we have only 256 PCIe buses in the system.

I'm even more hesitant about rearranging functions for VFIO devices. Here we're passing through a real PCI device that would be typically expected to occupy a slot, so I'd be concerned that userspace drivers that aren't expecting it to be a function in a multifunction slot will get confused.

I am always concerned when it comes to hot-plugging VFIO devices.
Regarding your specific concern, we are already re-arranging stuff, we take an SR-IOV Virtual function which is s PCI function belonging to a multi-function PCI device and we pass it to the VM as a non multi-function PCIe device.

However, if you think is too risky we can:

• have the VFIO devices use a whole pcie-root-port (VFs belonging to the same host PCI device can be grouped together)
• make it configurable and allow the admin to tweak kata configuration if there are issues.

[dgibson]

@dgibson

@marcel-apf using multiple root ports in a single slot via multifunction sounds good to me (though we will need Kata changes to accomodate it). I'm much more hesitant about placing different devices we want to attach as different functions of the same virtual PCI slot. For one thing, I thought QEMU's hotplug granularity was effectively whole slot, not just "plugged as a group must be unplugged as a group".

Indeed, there is room for only one PCI device per slot, but a PCI device may have several PCI functions and QEMU fully supports hot-plug of multi-function PCI devices. The hotplug operations are per slot/device and this is the reason we need to hotplug/hot-unplug all the PCI functions as a single operation.

Ah, I see you mean. That's probably a good end point to aim for, but I don't think it will be great for a first cut. At the moment Kata hotplugs the devices as they're discovered, to implement what you're suggesting we'd have to batch them up and collate them, which adds yet another significant reworking of the code to the several we already have here as prerequisites.

I see no reason why we shouldn't hotplug at least the virtio-pci-{type} as functions of the same device. Doing so will actually save PCI bus numbers, since each PCIe device uses a "whole" PCIe bus and we have only 256 PCIe buses in the system.

I'm even more hesitant about rearranging functions for VFIO devices. Here we're passing through a real PCI device that would be typically expected to occupy a slot, so I'd be concerned that userspace drivers that aren't expecting it to be a function in a multifunction slot will get confused.

I am always concerned when it comes to hot-plugging VFIO devices.
Regarding your specific concern, we are already re-arranging stuff, we take an SR-IOV Virtual function which is s PCI function belonging to a multi-function PCI device and we pass it to the VM as a non multi-function PCIe device.

Yeah, it's almost certainly going to be fine for SR-IOV virtual functions. But Kata doesn't really enforce or care whether we're passing a VF or a PF and it's the case of PFs that I'm more worried about.

However, if you think is too risky we can:

* have the VFIO devices use a whole pcie-root-port (VFs belonging to the same host PCI device can be grouped together)

* make it configurable and allow the admin to tweak kata configuration if there are issues.

[marcel-apf]

@dgibson @fidencio @devimc
I am sorry but I cannot make it to today's architecture meeting. Anyway, everything that is being discussed here is pending research before considering it.