docs: Document Intel Discrete GPUs usage with Kata #9084

amshinde · 2024-02-13T15:48:50Z

Document describes the steps needed to pass an entire Intel Discrete GPU as well a GPU SR-IOV interface to a Kata Container.

Fixes: #9083

mkbhanda · 2024-02-13T19:08:58Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+These include the Intel® Data Center GPU Max Series and Intel® Data Center GPU Flex Series.
+For integrated GPUs please refer to [Integrate-Intel-GPUs-with-Kata](../Intel-GPU-passthrough-and-Kata.md) 
+
+An Intel Discrate GPU can be passed to a Kata Containers Kata using GPU passthrough 


s/Containers Kata/Container

You've used "a", so singular and Kata repeated

How about " .. using one of GPU or SRIOV passthrough"

Shorter, clearly indicates only one of the technologies can be leveraged at any time.

mkbhanda · 2024-02-13T19:18:56Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+the VM to which it is assigned. The GPU is not shared among VMs.
+
+With SRIOV mode, it is possible to pass a Virtual GPU instance to a virtual machine.
+With this, multiple Virtual GPU instances carved out of a single GPU can be passed


s/With this, multiple Virtual GPU instances carved out of a single GPU can be passed
to multiple VMs at the same time allowing the GPU to be shared among them.
/s/With this, multiple Virtual GPU instances can be carved out of a single physical GPU and be passed to different VMs, allowing the GPU to be shared.

Some questions: does the VM have to be located on the same host where the GPU is attached? Any limit on the number of vGPUs once can carve out of a physical GPU? How does one slice a GPU, partition the GPU cores? Allocate all? Good to add a link to SRIOV GPU resource.

mkbhanda · 2024-02-13T19:20:11Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+| Technology | Description | Behaviour | Detail |
+|-|-|-|-|
+| Intel VT-d | GPU passthrough | Physical GPU assigned to a single VM | Direct GPU assignment to VM without limitation |
+| SRIOV | GPU sharing | Physical GPU shared by multiple VMs | SRIOV passthrough |


mkbhanda · 2024-02-13T19:20:55Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+## Host BIOS requirements
+
+Hardware such as Intel Max and Flex series, require larger PCI BARs. 


Do not need the ",".
Would be good to state what BAR is or are we assuming the user knows all this?
https://www.intel.com/content/www/us/en/support/articles/000090831/graphics.html

mkbhanda · 2024-02-13T19:26:25Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+Hardware such as Intel Max and Flex series, require larger PCI BARs. 
+
+For large BARs devices, MMIO mapping above 4G address space should be enabled in the PCI configuration of the BIOS.


grammar/reads funny "large BARs devices"

mkbhanda · 2024-02-13T23:12:32Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+1. Run the following to change the kernel command line using grub
+```
+sudo vim /etc/default/grub


Is it the same file on Ubuntu and other distributions?

mkbhanda · 2024-02-14T00:21:08Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+   ```
+
+   Run the previous command to determine the BDF for the GPU device on host.<br/>
+   From the previous output, PCI address `0000:29:00.0` is assigned to the hardware GPU device.<br/>


What is special about 29 versus 3a, 9a, or ca other than it is the lowest/listed first.

It is not. I just chose the first one. I have clarfied this now.

mkbhanda · 2024-02-14T00:59:53Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+4. Start Kata container with GPU device enabled:
+
+   ```
+   $ sudo ctr --debug run --runtime "io.containerd.kata.v2"  --device /dev/vfio/437  --rm -t  "docker.io/library/archlinux:latest" arch uname -r


In a Kubernetes environment is there some resource tracker that is watching out for which device nodes are available/free?

What happens if we are running a TDVM and try to connect a GPU?
An error? Or allows it with a warning say not-end-to-end-secure?

In a Kubernetes environment is there some resource tracker that is watching out for which device nodes are available/free?

A device plugin is for this purpose. I have demonstrated this with QAT but our GPU device plugin does not support GPU VF resources.

mythi · 2024-02-14T06:08:12Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+Use the following steps to pass an Intel discrete GPU  with Kata:
+
+1. Find the Bus-Device-Function (BDF) for GPU device:


How is the VF enablement done?

@ mythi the command mentioned earlier echo 4 | sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs creates the VFs. The out of tree kernel driver along with the kernel command line mentioned enables the VFs to be created.

Are the VFs automatically bound to vfio-pci?

jodh-intel

Thanks, @amshinde - A few initial (mostly formatting) comments. I'll try to take another look tomorrow...

jodh-intel · 2024-02-15T17:54:05Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+These include the Intel® Data Center GPU Max Series and Intel® Data Center GPU Flex Series.
+For integrated GPUs please refer to [Integrate-Intel-GPUs-with-Kata](../Intel-GPU-passthrough-and-Kata.md) 
+
+An Intel Discrate GPU can be passed to a Kata Container using GPU passthrough. 


Suggested change

An Intel Discrate GPU can be passed to a Kata Container using GPU passthrough.

An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough.

This typo is still present.

jodh-intel · 2024-02-15T17:55:57Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+For ubuntu:
+```
+sudo update-grub


Suggested change

sudo update-grub

$ sudo update-grub

jodh-intel · 2024-02-15T17:56:04Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+For Centos/RHEL:
+```
+sudo grub2-mkconfig -o /boot/grub2/grub.cfg 


Suggested change

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

jodh-intel · 2024-02-15T17:56:20Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+4. Reboot the system
+```
+sudo reboot


Suggested change

sudo reboot

$ sudo reboot

jodh-intel · 2024-02-15T17:57:30Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+configuration in the Kata `configuration.toml` file as shown below.
+
+```
+$ sudo sed -i -e 's/^# *\(hotplug_vfio_on_root_bus\).*=.*$/\1 = true/g' /usr/share/defaults/kata-containers/configuration.toml


The config may well be below /opt/kata/ if Kata is installed using kata-deploy / kata-manager.

jodh-intel · 2024-02-15T18:01:47Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+   Create SR-IOV interfaces for the GPU:
+   ```
+   $ echo 4 | sudo tee  /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs


Suggested change

$ echo 4 | sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs

$ echo 4 | sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs

jodh-intel · 2024-02-15T18:02:48Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+   ```
+   $ BDF="0000:3a:00:1"
+   $ readlink -e /sys/bus/pci/devices/$BDF/iommu_group


Suggested change

$ readlink -e /sys/bus/pci/devices/$BDF/iommu_group

$ readlink -e "/sys/bus/pci/devices/$BDF/iommu_group"

jodh-intel · 2024-02-15T18:03:12Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+   Now you can use the device node `/dev/vfio/437` in docker command line to pass
+   the VGPU to a Kata Container.
+
+4. Start Kata container with GPU device enabled:


Suggested change

4. Start Kata container with GPU device enabled:

4. Start a Kata Containers container with the GPU device enabled:

jodh-intel · 2024-02-15T18:04:27Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+5. Start a Kata container with GPU device:
+
+   ```
+   $ sudo ctr --debug run --runtime "io.containerd.kata.v2"  --device /dev/vfio/27  --rm -t  "docker.io/library/archlinux:latest" arch uname -r


Is it worth switching to quay.io now that docker.io has limits on it?

jodh-intel · 2024-02-15T18:04:40Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+

Nit: Extraneous blank line.

amshinde · 2024-02-22T17:09:05Z

@jodh-intel I have addressed your comments. Can you take another look?
@mkbhanda @mythi I have addressed your comments as well. Please have a look and let me know if you have any other comments/questions.

jodh-intel

Thanks, @amshinde - A few more comments...

jodh-intel · 2024-03-06T13:29:23Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+These include the Intel® Data Center GPU Max Series and Intel® Data Center GPU Flex Series.
+For integrated GPUs please refer to [Integrate-Intel-GPUs-with-Kata](../Intel-GPU-passthrough-and-Kata.md) 
+
+An Intel Discrate GPU can be passed to a Kata Container using GPU passthrough. 


This typo is still present.

jodh-intel · 2024-03-06T13:30:25Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+$ sudo apt install linux-headers-"$(UBUNTU_22.04_SERVER_KERNEL_VERSION) linux-image-unsigned-"$(UBUNTU_22.04_SERVER_KERNEL_VERSION)"
+$ make i915dkmsdeb-pkg
+```
+The above make command will create debain package in parent folder:  intel-i915-dkms_<release version>.<kernel-version>.deb


Suggested change

The above make command will create debain package in parent folder: intel-i915-dkms_<release version>.<kernel-version>.deb

The above `make` command will create a Debian package in the parent folder named: `intel-i915-dkms_<release version>.<kernel-version>.deb`.

jodh-intel · 2024-03-06T13:30:57Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+Below are the steps for installing the driver from source:
+```bash
+$ export I915_BRANCH="backport/main"
+$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git && cd intel-gpu-i915-backports/


I would suggest one command per line for clarity:

Suggested change

$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git && cd intel-gpu-i915-backports/

$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git

$ cd intel-gpu-i915-backports/

jodh-intel · 2024-03-06T13:36:49Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+The above make command will create debain package in parent folder:  intel-i915-dkms_<release version>.<kernel-version>.deb
+Install the package as:
+```bash
+$ sudo dpkg -i  intel-i915-dkms_<release version>.<kernel-version>.deb


Nit: Double space.

Suggested change

$ sudo dpkg -i intel-i915-dkms_<release version>.<kernel-version>.deb

$ sudo dpkg -i intel-i915-dkms_<release version>.<kernel-version>.deb

I also wonder if we should specify using apt-get or even just apt rather than the lower-level dpkg here.

Fixed the extra space. We are using dpkg for installing the local deb package that is generated.

jodh-intel · 2024-03-06T13:41:06Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+## Install and configure Kata Containers
+
+To use this feature, you need Kata version 1.3.0 or above.


It might be worth stating how you check the version you are running, so one of:

kata-runtime --version

kata-ctl version

containerd-shim-kata-v2 --version

bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/main/utils/kata-manager.sh) -l

In fact, for bonus points, it might be worth adding a doc or doc section somewhere else with these details and just referencing that here.

jodh-intel · 2024-03-06T13:52:04Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+```bash
+$ sudo apt update
+$ sudo apt install -y gpg-agent wget
+$ wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \


Can you use curl rather than wget.

I think use of wget should be ok here. I would also rather use wget here to keep in line with the installation instructions on the Intel site.

jodh-intel · 2024-03-06T13:53:18Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+3. Update grub as per OS distribution:
+
+For ubuntu:


Suggested change

For ubuntu:

For Ubuntu:

jodh-intel · 2024-03-06T13:53:42Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+$ sudo update-grub
+```
+
+For Centos/RHEL:


Suggested change

For Centos/RHEL:

For CentOS/RHEL:

jodh-intel · 2024-03-06T13:57:48Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+$ sudo apt install -y gpg-agent wget
+$ wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
+    sudo gpg --dearmor --output /usr/share/keyrings/intel-graphics.gpg
+$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${VERSION_CODENAME}/lts/2350 unified" | \


It may make sense to add a note at the version top of the document stating that your system must have an Intel x86_64 CPU. I know, I know! ... But I suspect someone will still try it! 😄

jodh-intel · 2024-03-06T14:01:02Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+$ export I915_BRANCH="backport/main"
+$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git && cd intel-gpu-i915-backports/
+$ sudo apt install dkms make debhelper devscripts build-essential flex bison mawk
+$ sudo apt install linux-headers-"$(UBUNTU_22.04_SERVER_KERNEL_VERSION) linux-image-unsigned-"$(UBUNTU_22.04_SERVER_KERNEL_VERSION)"


These variable names are not valid. I understand why we're doing this, but I still think it would be clearer to:

State at the top of the doc that we're assuming you are running on a Ubuntu 22.04 LTS system.

Change this code to reference the output of uname -r.

State here in a note that if you are not running on that version of Ubuntu, you will need to manually determine the latest 22.04 kernel version.

mythi · 2024-03-07T08:01:41Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+In Intel GPU pass-through mode, an entire physical GPU is directly assigned to one VM. 
+In this mode of operation, the GPU is accessed exclusively by the Intel driver running in
+the VM to which it is assigned. The GPU is not shared among VMs.


Is this a likely setup that is worth documenting?

We have covered this case in the past for Intel Integrated and Nvidia GPUs. Worth mentioning for completeness. I also feel this case will become more important in the Confidential Containers scenario.

amshinde · 2024-03-12T19:45:26Z

@jodh-intel I have addressed your comments. Please take a look.

jodh-intel

Thanks, @amshinde - I've found a few more points (mostly nits), but please tal.

jodh-intel · 2024-03-13T08:04:50Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough. 
+as well as SRIOV passthrough.


Detached sentence. Do we need something like:

Suggested change

An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough.

as well as SRIOV passthrough.

An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough, or SRIOV passthrough.

jodh-intel · 2024-03-13T08:06:35Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+| Technology | Description |
+|-|-|
+| GPU passthrough | Physical GPU assigned to a single VM |
+| SR-IOV passthrough | Physical GPU shared by multiple VMs |


This doc contains a mixture of "SRIOV" and "SR-IOV". Can you select one term and use it consistently.

jodh-intel · 2024-03-13T08:10:19Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+With this, multiple Virtual GPU instances can be carved out of a single physical GPU 
+and be passed to different VMs, allowing the GPU to be shared.
+
+| Technology | Description |


It might be worth adding some extra columns showing the proportion of the GPU that is available in the different contexts. Something like:

Technology Proportion of GPU shared to VM Proportion of GPU accessible to host Description

GPU passthrough 100% 0% Physical GPU assigned to a single VM

SR-IOV passthrough varies varies Physical GPU shared by multiple VMs

If so, it does raise the question, "What is the minimum GPU % I can pass to a single VM when using SR-IOV?" so it might be worth stating that as a note.

I think adding the extra column is confusing in this case. Since we are dealing with passthrough, we should be conncerned about the proportion shared with the VM.

jodh-intel · 2024-03-13T08:11:33Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+- Intel® Data Center GPU Max Series (`Ponte Vecchio`)
+- Intel® Data Center GPU Flex Series (`Arctic Sound-M`)
+- Intel® Data Center GPU Arc Series 
+


Is it worth adding a note here explaining how the user can query their system to determine if they have a recommended Intel GPU?

jodh-intel · 2024-03-13T08:11:54Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+- Intel® Data Center GPU Flex Series (`Arctic Sound-M`)
+- Intel® Data Center GPU Arc Series 
+
+The following steps outline the workflow for using an Intel Graphics device with Kata.


Nit:

Suggested change

The following steps outline the workflow for using an Intel Graphics device with Kata.

The following steps outline the workflow for using an Intel Graphics device with Kata Containers.

jodh-intel · 2024-03-13T08:19:24Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+For support on other distributions, please refer to [DGPU-docs](https://dgpu-docs.intel.com/driver/installation.html)
+
+You can also install the driver from source which is maintained at [intel-gpu-i915-backports](https://github.com/intel-gpu/intel-gpu-i915-backports)
+Detailed instructions for reference can be found at: https://github.com/intel-gpu/intel-gpu-i915-backports/blob/backport/main/docs/README_ubuntu.md


Nit:

Suggested change

Detailed instructions for reference can be found at: https://github.com/intel-gpu/intel-gpu-i915-backports/blob/backport/main/docs/README_ubuntu.md

Detailed instructions for reference can be found at: https://github.com/intel-gpu/intel-gpu-i915-backports/blob/backport/main/docs/README_ubuntu.md.

jodh-intel · 2024-03-13T08:19:58Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+$ sudo apt install dkms make debhelper devscripts build-essential flex bison mawk
+$ sudo apt install linux-headers-"$(uname -r) linux-image-unsigned-"$(uname -r)"


Nit: For consistency with previous commands:

Suggested change

$ sudo apt install dkms make debhelper devscripts build-essential flex bison mawk

$ sudo apt install linux-headers-"$(uname -r) linux-image-unsigned-"$(uname -r)"

$ sudo apt install -y dkms make debhelper devscripts build-essential flex bison mawk

$ sudo apt install -y linux-headers-"$(uname -r) linux-image-unsigned-"$(uname -r)"

jodh-intel · 2024-03-13T08:20:55Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+The above make command will create debain package in parent folder:  intel-i915-dkms_<release version>.<kernel-version>.deb
+Install the package as:
+```bash
+$ sudo dpkg -i intel-i915-dkms_<release version>.<kernel-version>.deb


We should add a warning pointing out that this package won't be automatically updated since it's been manually installed.

I dont think we really this here, we are installing local package using dpkg, so the user should be well aware of this.

jodh-intel · 2024-03-13T08:22:47Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+$ sudo reboot
+```
+
+Additionally, verify that the following kernel configs are enabled on your host kernel:


Nit:

Suggested change

Additionally, verify that the following kernel configs are enabled on your host kernel:

Additionally, verify that the following kernel configs are enabled for your host kernel:

jodh-intel · 2024-03-13T08:23:13Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+1. Run the following to change the kernel command line using grub
+```bash
+sudo vim /etc/default/grub


Nit: Consistency: missing prompt:

Suggested change

sudo vim /etc/default/grub

$ sudo vim /etc/default/grub

jodh-intel

Thanks, @amshinde.

lgtm

dborquez

Thank you @amshinde , just few comments:

dborquez · 2024-03-22T14:04:03Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+Your host kernel needs to be booted with `intel_iommu=on` and `i915.enable_iaf=0` on the kernel command
+line.
+
+1. Run the following to change the kernel command line using grub


Suggested change

1. Run the following to change the kernel command line using grub

1. Run the following to change the kernel command line using grub:

dborquez · 2024-03-22T14:35:49Z

docs/use-cases/Intel-Discrete-GPU-passthrough-and-Kata.md

+
+   Create SR-IOV interfaces for the GPU:
+   ```bash
+   $ echo 4 | sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs


Suggested change

$ echo 4 | sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs

$ echo 4 | sudo tee /sys/bus/pci/devices/$BDF/sriov_numvfs

dborquez

lgtm, thank you @amshinde

amshinde · 2024-03-26T17:20:46Z

/test

amshinde · 2024-04-08T20:47:33Z

/test

Document describes the steps needed to pass an entire Intel Discrete GPU as well a GPU SR-IOV interface to a Kata Container. Fixes: kata-containers#9083 Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>

Configuration file for qemu with runtime-rs was recently renamed. Doc contains name for old file. This was somehow not caught in the CI earlier. Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>

Add missing words to spell-check dictionaries Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>

amshinde · 2024-04-16T18:54:34Z

/test

amshinde requested a review from a team as a code owner February 13, 2024 15:48

katacontainersbot added the size/large Task of significant size label Feb 13, 2024

mkbhanda reviewed Feb 14, 2024

View reviewed changes

mythi reviewed Feb 14, 2024

View reviewed changes

amshinde force-pushed the document-intel-gpu-vfio branch from 0264b5d to ec3b547 Compare February 15, 2024 09:12

jodh-intel reviewed Feb 15, 2024

View reviewed changes

amshinde force-pushed the document-intel-gpu-vfio branch 2 times, most recently from f4e9260 to 3daae0c Compare February 22, 2024 17:07

jodh-intel reviewed Mar 6, 2024

View reviewed changes

mythi reviewed Mar 7, 2024

View reviewed changes

amshinde force-pushed the document-intel-gpu-vfio branch from 3daae0c to c0fbc6e Compare March 12, 2024 19:04

jodh-intel reviewed Mar 13, 2024

View reviewed changes

amshinde force-pushed the document-intel-gpu-vfio branch from c0fbc6e to bbdf21b Compare March 18, 2024 17:38

jodh-intel approved these changes Mar 21, 2024

View reviewed changes

amshinde force-pushed the document-intel-gpu-vfio branch from bbdf21b to c49df68 Compare March 21, 2024 23:57

dborquez requested changes Mar 22, 2024

View reviewed changes

amshinde force-pushed the document-intel-gpu-vfio branch from c49df68 to 52721d3 Compare March 22, 2024 17:24

dborquez approved these changes Mar 22, 2024

View reviewed changes

amshinde added the ok-to-test label Mar 26, 2024

amshinde force-pushed the document-intel-gpu-vfio branch from 52721d3 to 25fe017 Compare April 8, 2024 20:44

amshinde force-pushed the document-intel-gpu-vfio branch 2 times, most recently from 11084ae to b5016e5 Compare April 16, 2024 17:45

Archana Shinde and others added 3 commits April 16, 2024 11:50

docs: Document Intel Discrete GPUs usage with Kata

87f0097

Document describes the steps needed to pass an entire Intel Discrete GPU as well a GPU SR-IOV interface to a Kata Container. Fixes: kata-containers#9083 Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>

static-checks: Rename file in doc to make static checks happy

6f97dc1

Configuration file for qemu with runtime-rs was recently renamed. Doc contains name for old file. This was somehow not caught in the CI earlier. Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>

spell-check: Add missing words to spell-check

973a153

Add missing words to spell-check dictionaries Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>

amshinde force-pushed the document-intel-gpu-vfio branch from b5016e5 to 973a153 Compare April 16, 2024 18:50

amshinde merged commit af3b19e into kata-containers:main Apr 16, 2024
295 of 304 checks passed


		## Host BIOS requirements

		Hardware such as Intel Max and Flex series, require larger PCI BARs.


		Hardware such as Intel Max and Flex series, require larger PCI BARs.

		For large BARs devices, MMIO mapping above 4G address space should be enabled in the PCI configuration of the BIOS.


		Use the following steps to pass an Intel discrete GPU with Kata:

		1. Find the Bus-Device-Function (BDF) for GPU device:

	An Intel Discrate GPU can be passed to a Kata Container using GPU passthrough.
	An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough.

	sudo grub2-mkconfig -o /boot/grub2/grub.cfg
	$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

	$ echo 4 \| sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs
	$ echo 4 \| sudo tee /sys/bus/pci/devices/0000\:3a\:00.0/sriov_numvfs

	$ readlink -e /sys/bus/pci/devices/$BDF/iommu_group
	$ readlink -e "/sys/bus/pci/devices/$BDF/iommu_group"

	4. Start Kata container with GPU device enabled:
	4. Start a Kata Containers container with the GPU device enabled:

	The above make command will create debain package in parent folder: intel-i915-dkms_<release version>.<kernel-version>.deb
	The above `make` command will create a Debian package in the parent folder named: `intel-i915-dkms_<release version>.<kernel-version>.deb`.

	$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git && cd intel-gpu-i915-backports/
	$ git clone -b ${I915_BRANCH} --depth 1 https://github.com/intel-gpu/intel-gpu-i915-backports.git
	$ cd intel-gpu-i915-backports/

	$ sudo dpkg -i intel-i915-dkms_<release version>.<kernel-version>.deb
	$ sudo dpkg -i intel-i915-dkms_<release version>.<kernel-version>.deb


		## Install and configure Kata Containers

		To use this feature, you need Kata version 1.3.0 or above.

		An Intel Discrete GPU can be passed to a Kata Container using GPU passthrough.
		as well as SRIOV passthrough.

Technology	Proportion of GPU shared to VM	Proportion of GPU accessible to host	Description
GPU passthrough	100%	0%	Physical GPU assigned to a single VM
SR-IOV passthrough	varies	varies	Physical GPU shared by multiple VMs

	The following steps outline the workflow for using an Intel Graphics device with Kata.
	The following steps outline the workflow for using an Intel Graphics device with Kata Containers.

	Detailed instructions for reference can be found at: https://github.com/intel-gpu/intel-gpu-i915-backports/blob/backport/main/docs/README_ubuntu.md
	Detailed instructions for reference can be found at: https://github.com/intel-gpu/intel-gpu-i915-backports/blob/backport/main/docs/README_ubuntu.md.

		$ sudo apt install dkms make debhelper devscripts build-essential flex bison mawk
		$ sudo apt install linux-headers-"$(uname -r) linux-image-unsigned-"$(uname -r)"

	Additionally, verify that the following kernel configs are enabled on your host kernel:
	Additionally, verify that the following kernel configs are enabled for your host kernel:

	1. Run the following to change the kernel command line using grub
	1. Run the following to change the kernel command line using grub:

docs: Document Intel Discrete GPUs usage with Kata #9084

docs: Document Intel Discrete GPUs usage with Kata #9084

Conversation

amshinde commented Feb 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amshinde commented Feb 22, 2024

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel Mar 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amshinde commented Mar 12, 2024

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel left a comment

jodh-intel Mar 6, 2024 •

edited