GPU Operator with KubeVirt - Node Configuration

[doronkg]

Hi! The following excerpt is taken from the official documentation on GPU Operator with KubeVirt docs:

The GPU Operator can now be configured to deploy different software components on worker nodes depending on what GPU workload is configured to run on those nodes. Consider the following example.

Node A is configured to run containers.
Node B is configured to run virtual machines with Passthrough GPU.
Node C is configured to run virtual machines with vGPU.

We provide GPU capabilities through OpenShift, utilizing the NVIDIA GPU Operator. We aim to enable both GPUs for containers and vGPUs for VMs on the same cluster, ideally on the same nodes.

According to the latest NVIDIA GPU Operator release notes, it seems like this isn't feasible at all, as:

The NVIDIA GPU Operator can only be used to deploy a single NVIDIA GPU Driver type and version. The NVIDIA vGPU and Data Center GPU Driver cannot be used within the same cluster.

This discrepancy in the documentation raises several questions:

1. Is it, or is it not possible to configure GPU access for both containers and VMs on the same cluster or node?
   • If not, is this limitation inherent to the software, or could it be overcome through operator configuration workarounds?
   • If not, are there any plans to introduce dual configuration support for the same cluster/node in the future?
     • If the answer is yes, is it under open discussion & development that accepts contributions?
2. Can vGPUs be utilized through containers, thereby allowing both methods to be employed using a single configuration?

[cdesiniotis]

According to the latest NVIDIA GPU Operator release notes, it seems like this isn't feasible at all, as:
The NVIDIA GPU Operator can only be used to deploy a single NVIDIA GPU Driver type and version. The NVIDIA vGPU and Data Center GPU Driver cannot be used within the same cluster.

The limitation you quoted only applies to container workloads. That is, you cannot deploy both the data center driver and vGPU guest driver in the same cluster. We hope to remove this limitation in a future release.

However (as I will discuss below), you can install the data center driver OR the vGPU guest driver on some nodes (which are used by container workloads), and have other nodes configured with the vGPU Host driver or vfio-pci driver (which support running VM workloads).

Is it, or is it not possible to configure GPU access for both containers and VMs on the same cluster or node?

You can run multiple types of workloads in the same cluster, but not multiple on the same node. You can use the node label nvidia.com/gpu.workload.config={container, vm-passthrough, vm-vgpu} to configure the type of workload you want to support on a particular node. For example, labeling a node with nvidia.com/gpu.workload.config=vm-passthrough will prepare all GPUs on the node for passthrough -- all GPUs get bound to the vfio-pci driver and the device-plugin running on that node will advertise these resources as passthrough devices.

There are technical challenges with supporting a mix of workload types on the same node, especially with the current device plugin API. That said, the future of GPU support will be with a relatively new Kubernetes feature called Dynamic Resource Allocation. This will naturally be much easier to implement and support with this model.

More information here: https://docs.google.com/document/u/1/d/1BNWqgx_SmZDi-va_V31v3DnuVwYnF2EmN7D-O_fB6Oo/mobilebasic

Can vGPUs be utilized through containers, thereby allowing both methods to be employed using a single configuration?

Yes, you can run containers with vGPU. This requires configuring the GPU Operator to use the vGPU guest driver instead of the data center driver. You can refer to our documentation here:
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-vgpu.html
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-nvaie.html

[doronkg]

@cdesiniotis please correct me if I'm wrong and I would like to have your take on this.

As I understand it, there're 2 methods to enable GPU access for either VM or container workloads:

VMs:

1. pGPU - PCI passthrough, a direct bind between a GPU card and a VM
2. vGPU - virtual GPU slices

Containers:

1. GPU - allocation of an entire GPU card
2. vGPU - virtual GPU slices

---

Therefore, there're 4 options to enable GPU access for both VMs and containers in the same cluster (but not the same nodes):

	VM-pGPU	VM-vGPU	Container-GPU	Container-vGPU
Option A	✔️		✔️
Option B	✔️			✔️
Option C		✔️	✔️
Option D		✔️		✔️

---

I couldn't find references about vGPU for containers in the docs, so the following questions were raised:

1. How does vGPU differ from regular time-slicing?
2. Does vGPU for containers require a different license from vGPU for VMs?
3. Does vGPU for containers guarantee allocation and memory isolation?
4. Does vGPU for containers configure the GPU in WDDM or TCC mode?
5. DCGM exporter doesn't exist when configuring vGPU for VMs, same for containers?

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[cdesiniotis]

@doronkg "vGPU for containers" is not really a concept. Software running within a guest VM just sees a GPU (i.e. it does not know if it is a vGPU or a passthrough device). If you want multiple containers to share access to the same GPU, you can leverage MIG (hardware partitioning) or time-slicing / MPS (software partitioning). These sharing strategies are covered in the following resources:
https://developer.nvidia.com/blog/improving-gpu-utilization-in-kubernetes/
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html
https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html

I am closing this issue. Re-open this issue or file a new one if you have additional questions.