Skip to content

GVTd_Setup_Guide

Terrence Xu edited this page Dec 2, 2021 · 11 revisions

Contents

1 Introduction

Intel GVT-d is one flavor of graphics virtualization approaches also names as Intel-Graphics-Device pass-through feature. It is based on Intel VT-d technology and some special graphics related configuration than NIC pass-through. This flavor allows direct assignment of an entire GPU’s prowess to a single user, passing the native driver capabilities through the hypervisor without any limitations.

2 System Requirements

2.1 Operating System Requirements

Ubuntu 20.04 has been fully validated as host, other Linux operating system like RHEL with 4.X kernel is also OK.

2.2 Hardware Requirements

For client platforms, from 5th to 11th is supported. For server platforms, E3_v4, E3_v5, or E3_v6 Xeon Processor Graphics is validated.

2.3 Library Dependence

Use Ubuntu as an example, there are some software packages needed by host environment setup, as below:

apt-get install git libfdt-dev libpixman-1-dev libssl-dev vim socat libsdl1.2-dev libspice-server-dev autoconf libtool xtightvncviewer tightvncserver x11vnc libsdl1.2-dev uuid-runtime uuid uml-utilities bridge-utils python-dev liblzma-dev libc6-dev

3 Host Environment Setup

3.1 Source Repositories

Kernel: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
QEMU: git://git.qemu.org/qemu.git
Xen : git://xenbits.xen.org/xen.git

3.2 Build Kernel

git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git

cd linux-stable

git checkout master

With the default kernel, Linux VM will trigger GPU hang/GPU reset while running 2D /3D application, which is a known issue with hypervisor KVM, so we can add the disable stolen memory patch to resolve this problem (If you don't want to add this patch to kernel, you also can enable stolen memory patch in Qemu side, as section 3.3).

echo ""|make oldconfig

make -j8 && make modules_install && make install

3.3 Build Qemu

Please follow chapter 3.3.1 for GVT-d KVM setup and chapter 3.3.2 for GVT-d Xen setup.

3.3.1 Build Qemu for GVT-d KVM

git clone git://git.qemu.org/qemu.git

cd qemu

git checkout master

git revert 93587e3af3a // enable stolen memory patch if you didn't do it in kernel side

./configure --prefix=/usr \

--target-list=x86_64-softmmu

make -j8

make install

3.3.2 Build Qemu & Xen for GVT-d Xen

git clone git://xenbits.xen.org/xen.git

cd xen

git checkout master

./configure

a. In order to run GVT-d well in 5th Gen or E3_v4 Xeon platform, we need to edit the rombios file with below changes:

tools/firmware/rombios/rombios.c
mov ax, #0xc780
call rom_scan
+ mov ax, #0x03
+ int #0x10
+
call _print_bios_banner;

make xen -j8

make tools –j8

b. In order to GVT-d run well in 6th / 7th Gen or E3_v5 / E3_v6 platform, we need to edit the qemu-xen-tradtional-dir file with below changes:

tools/qemu-xen-traditional-dir/hw/pt-graphics.c
+memset(bios, 0 ,bios_size);
cpu_physical_memory_rw(0xc0000, bios, bios_size, 1);

Then you need to build tools again.

make tools -j8

make install-xen

make install-tools

3.4 Grub Update

You need to add the GVT-d KVM and GVT-d Xen corresponding menu entry in your grub.cfg, or you can add it to /etc/default/grub is better. (Since grub.cfg will be auto regenerated by Linux distro version upgrade or kernel package update).

For GVT-d KVM, Make sure the "intel_iommu=on" exist in your menuentry, below is an example on ubuntu:

menuentry 'GVT-d KVM' --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
    recordfail
    load_video
    gfxmode $linux_gfx_mode
    insmod gzio
    insmod part_msdos
    insmod ext2
    set root='hd0,msdos1'
    if [ x$feature_platform_search_hint = xy ]; then
        search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  106518cd-fb74-4048-81de-3ecee7ad74d4
    else
        search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
    fi
        linux   /boot/vmlinuz-4.17.0+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel log_buf_len=128M console=ttyS0,115200,8n1 kvm.ignore_msrs=1 intel_iommu=on
        initrd  /boot/initrd.img-4.17.0+
}

NOTICE: Since some Windows guest 3rd party applications/tools (like GPU-Z / Passmark9.0) will trigger MSR read/write directly if it accesses the unhandled MSR register, the guest will trigger BSOD soon. So we added the "kvm.ignore_msrs=1" into grub for workaround.

For GVT-d Xen, make sure the "modprobe.blacklist=i915 xen-pciback.passthrough=1 xen-pciback.hide=(00:02.0)" exist in your menuentry, below is an example on ubuntu:

menuentry 'GVT-d Xen' - --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
	recordfail
	load_video
	gfxmode $linux_gfx_mode
	insmod gzio
	insmod part_msdos
	insmod ext2
	set root='hd0,msdos1'
	if [ x$feature_platform_search_hint = xy ]; then
		search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  106518cd-fb74-4048-81de-3ecee7ad74d4
	else
		search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
	fi
	multiboot   /boot/xen.gz dom0_max_vcpus=2 dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all msi=1 conring_size=4M  console=com1 com1=115200,8n1 sync_console
	module   /boot/vmlinuz-4.17.0+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel intel_iommu=on log_buf_len=128M console=hvc0  modprobe.blacklist=i915 xen-pciback.passthrough=1 xen-pciback.hide=(00:02.0)
	module  /boot/initrd.img-4.17.0+
}

4 Guest Setup

4.1 Linux Guest Setup

4.1.1 Linux Kernel Support

You can prepare any version Ubuntu or any other Linux guest OS image manually. Then you can change the guest kernel to the same kernel as host, you also can download the stable version from https://www.kernel.org/ or use Ubuntu18.04 guest OS kernel directly.

4.1.2 Linux Intel GFX Card Support

  • We need to make the special X11 configuration change for Linux guests to use the Intel graphics card as a display card.

  • The emulated graphics card (like "QXL", "Cirrus")not support OpenCL / OpenGL hardware acceleration, we must make this change.

  • Here we use Ubuntu as an example, we save the following section at /etc/X11/xorg.conf file:

      Section "Device"
              Identifier  "intel"
              Driver      "intel"
              BusID       "PCI:0:4:0"  # Sample: "PCI:0:2:0"
      EndSection
      Section "Screen"
              Identifier  "intel"
              Device "intel"
      EndSection
    

BTW: The PCI ID was chosen by looking at the output of the "lspci" command.

root@gvtd-host:~# lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]

00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]

00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)

00:02.0 VGA compatible controller: Cirrus Logic GD 5446

00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)

00:04.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 0a)

NOTICE: After you change the display card to the Intel graphics card, Qemu Windows will show black screen, you only can see the guest desktop by a remote protocol.

4.1.3 Linux Remote Protocol Support

We suggest using the X11VNC as the remote protocol to validation which has been validated by us.

Use Ubuntu as an example, to support X11VNC, you must install the xhost in guest first, the xhost program is used to add and delete host names or user names to the list allowed to make connections to the X server, you must make some configuration for lightdm, the example as below:

root@gvtd-host:~# cat /etc/profile.d/lightdm_display.sh
#!/bin/bash
    export DISPLAY=:0

    if [ ! -z "$DISPLAY" ]; then
        xhost +local:
    fi

root@gvtd-host:~# chmod +x /etc/profile.d/lightdm_display.sh

root@gvtd-host:~# cat /etc/lightdm/lightdm.conf
    [SeatDefaults]
    greeter-session=unity-greeter
    user-session=ubuntu
    greeter-setup-script=/etc/profile.d/lightdm_display.sh
    session-setup-script=/etc/profile.d/lightdm_display.sh

After you complete the lightdm configuration, you can run the X11VNC service in guest by following commands or similar commands:

x11vnc -display :0 -forever -bg -repeat -nowf -o ~/.vnc/x11vnc.log -rfbport 5900

Then connect to the guest by the host with guest X support by typing the following command:

vncviewer $guestIP

NOTICE: $guestIP means guest IP address, you can get the guest IP address by "nmap" / "arp-scan" or any other network-related tools in the host.

Except for X11VNC, there are some other alternatives, which were not validated by us, you also can use it as a remote protocol for a try:

4.2 Windows Guest Setup

4.2.1 Windows Version Support

Win7-32 / Win7-64 / Win8.1-64 /Win10-RS4-64 are validated.

4.2.2 Windows GFX Driver Support

  • After you boot up Windows guest, you can see two graphics card in "Device manager"

  • The virtual GFX card "QXL" or "Cirrus"

  • The Intel GFX card

  • Then you must install the Intel GFX driver for the Intel Graphics card to support OpenCL / OpenGL / DirectX9 / DirectX10/ DirectX11 / DirectX12 hardware acceleration .

  • You can get the Windows driver from graphics drivers section in the Intel website. Depending on your Intel processor, the latest available driver versions as below:

    Kabylake & Skylake & Coffeelake & newer platform: Win10-64: 30.0.100.9667

    Skylake Win7-32 / Win7-64 / Win8.1-64: 15.45.31.5127

    Broadwell: 15.40.45.5126

NOTICE: After you install Windows GFX driver, you can see guest desktop in both QEMU and remote protocol. QEMU displays via the emulated driver and the remote protocol displays via the Intel GFX driver. BTW, There is a known issue that some 3D workload cannot run with hardware acceleration while both GFX card is available, so suggest to disable the emulated GFX card in "Device Manager" after you make sure the Intel GFX card is working.

4.2.3 Windows Remote Protocol Support

We suggest using Tightvnc as the remote protocol to validation which has been validated by us.

Then connect to the guest by the host with guest X support by typing the following command:

vncviewer $guestIP

NOTICE: $guestIP means guest IP address, you can get the guest IP address by "nmap" / "arp-scan" or any other network-related tools in the host.

All Windows Remote protocols are supported, such as TightVNC, HP RGS, RDP.

Except for Tightvnc, there are some other alternatives, which were not validated by us, you also can use it as a remote protocol for a try:

5 Basic Usage

5.1 Preparing a graphic card device for passthrough for GVT-d KVM

root@gvtd-host:~# modprobe vfio

root@gvtd-host:~# modprobe vfio_pci

root@gvtd-host:~# lspci -D -nn

0000:00:00.0 Host bridge [0600]: Intel Corporation Broadwell-U Host Bridge - DMI [8086:1614] (rev 06)

0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Broadwell-U Integrated Graphics [8086:1622] (rev 08)

Thus it can be seen, VGA device (0000:00:02.0,vendor & device ID 8086:1622)

root@gvtd-host:~# echo 0000:00:02.0 > /sys/bus/pci/devices/0000:00:02.0/driver/unbind

root@gvtd-host:~# echo 8086 1622 > /sys/bus/pci/drivers/vfio-pci/new_id

5.2 Preparing a graphic card device for passthrough for GVT-d Xen

root@gvtd-host:~# xl pci-assignable-add 00:02.0

root@gvtd-host:~# xl pci-assignable-list 0000:00:02.0

5.3 Network Bridge

For GVT-d KVM, in order to let guests using the same network segment as host, you must create the network bridge. Besides, in order to let Qemu using the network bridge you created, you also need to create a script as below (We saved it at /etc/qemu-ifup):

For details you can reference the the "Configuring Guest Networking" wiki page http://www.linux-kvm.org/page/Networking

For GVT-d Xen you can just create the network bridge by yourself.

5.6 Create VM

5.6.1 Create GVT-d KVM VM

We have two methods to create VM, one is UPT mode (guest cannot local display in the monitor), the other is legacy mode (guest can local display in monitor), the "legacy mode" was recommended.

The example of creating VM script as below :

UPT mode:

#! /bin/bash -x
    /usr/bin/qemu-system-x86_64 \
    -enable-kvm \ 
    -m 2048 \
    -smp 2 \
    -cpu host \
    -hda /home/img/win10-64.qcow2 \
    -vga cirrus \
    -device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x6 \
    -usb -usbdevice tablet \
    -net nic,macaddr=00:16:3e:60:0a:50 -net tap,script=/etc/qemu-ifup \
    -daemonize

Legacy mode:

#! /bin/bash -x
    /usr/bin/qemu-system-x86_64 \
    -enable-kvm \
    -m 2048 \
    -smp 2 \
    -cpu host \
    -hda /home/img/win10-64.qcow2 \
    -vga none -nographic \
    -device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,bus=pci.0,addr=0x2,x-igd-opregion=on \
    -usb -usbdevice tablet \
    -vnc :2 \
    -net nic,macaddr=00:16:3e:60:0a:50,addr=0x3 -net tap,script=/etc/qemu-ifup \
    -daemonize

5.6.2 Create GVT-d Xen VM

You can create hvm configuration file by modifying /etc/xen/xlexample.hvm like below:

builder = "hvm"
name = "HVM_IGD_PASSTHROUGH"
memory = 2048
vcpus=4
vif = [ 'type=ioemu, mac=00:16:3e:30:e0:6c, bridge=xenbr0' ]
disk = [ '/home/img/win10-red-stone.img,raw,hda,rw' ]
device_model_override = '/usr/local/lib/xen/bin/qemu-dm'
device_model_version = 'qemu-xen-traditional'

sdl=0
vnc=1

# default rdm_mem_boundary is 2G
rdm_mem_boundary=700
rdm="strategy=host,policy=strict"
pci=[ "00:02.0,rdm_policy=strict" ]
gfx_passthru=1
usb=1
usbdevice='tablet'

After that, you can create XenGT VM via the below command:

xl create xlexample.hvm

6 Features Supported

6.1 Highlights Features

  • General:
    • Supported Linux distributions (64bit): Ubuntu, RHEL, SUSE, etc
    • Supported Windows guest: Windows7 (64bit), Windows8.1 (64bit), Windows10 RedStone (64bit)
    • Single display with resolution up to 1920x1080p
  • 3D/2D:
    • OpenGL4.5 for Linux guest
    • OpenGL4.4 for Windows guest
    • DirectX9, 10, 11, 12 for Windows guest
  • Compute:
    • OpenCL2.0 for Windows/Linux guest
  • Media:
    • Intel Media Service Studio (2015R5)
  • Remote display protocol:
    • X11VNC for Linux guest
    • TightVNC, HP RGS, RDP for Windows guest

6.2 Validated Benchmarks

  • 3D / 2D:
    • 3Dmark06, 3Dmark11, WebGL, Passmark, Unigine-tropics, Unigine-heaven, Phoronix 3D, Cairo 2D, etc
  • Media:
    • Multiple thread decode/transcode for JPEG, MPEG2, H264, HEVC, VC1, VP8, etc
  • OpenCL:
    • LuxMark, Beignet