GVTg_Setup_Guide

Terrence Xu edited this page Feb 12, 2019 · 52 revisions

Contents

1 Introduction

Intel GVT-g is a full GPU virtualization solution with mediated pass-through (VFIO mediated device framework based), starting from 5th generation Intel Core(TM) processors with Intel Graphics processors. GVT-g supports both Xen and KVM (a.k.a XenGT & a.k.a KVMGT). A virtual GPU instance is maintained for each VM, with part of performance critical resources directly assigned. The capability of running native graphics driver inside a VM, without hypervisor intervention in performance critical paths, achieves a good balance among performance, feature, and sharing capability.

2 System Requirements

2.1 Operating System Requirements

Ubuntu 18.04. has been fully validated as host, other Linux operating system like RHEL / Fedora are also OK.

2.2 Hardware Requirements

For client platforms, 5th, 6th, 7th or 7th SoC Generation Intel® Core Processor Graphics is required. For server platforms, E3_v4, E3_v5 or E3_v6 Xeon Processor Graphics is required.

2.3 Library Dependence

Use Ubuntu as an example, there are some software package needed by host environment setup, as below:

apt-get install git libfdt-dev libpixman-1-dev libssl-dev vim socat libsdl1.2-dev libspice-server-dev autoconf libtool xtightvncviewer tightvncserver x11vnc libsdl1.2-dev uuid-runtime uuid uml-utilities bridge-utils python-dev liblzma-dev libc6-dev libegl1-mesa-dev libepoxy-dev libdrm-dev libgbm-dev libaio-dev libusb-1.0.0-dev

3 Host Environment Setup

3.1 Source Repositories

GVT-g kernel: https://github.com/intel/gvt-linux/  (branch: gvt-stable-4.17)
QEMU: https://github.com/intel/igvtg-qemu  (branch: stable-2.12.0)
Xen : https://github.com/intel/igvtg-xen (branch: xengt-stable-4.10)

NOTICE: For KVMGT, you also can use the current upstream Linux kernel and QEMU directly since all the enabling patches have been upstreamed.

For XenGT, you must use the repositories we provided.

3.2 Build Kernel

3.2.1 Build the initrd (initial ramdisk)

Use Ubuntu as example, modify /etc/initramfs-tools/modules, like below:

kvmgt

xengt

vfio-iommu-type1

vfio-mdev

For Fedora or RHEL you should make the similar changes in "dracut".

3.2.2 Build Kernel Source

git clone https://github.com/intel/gvt-linux.git

cd gvt-linux

git checkout gvt-stable-4.17

echo ""|make oldconfig

Then make sure to enable CONFIG_DRM_I915_GVT, CONFIG_DRM_I915_GVT_KVMGT and CONFIG_DRM_I915_GVT_XENGT in ".config", which depends on CONFIG_VFIO_MDEV and CONFIG_VFIO_MDEV_DEVICE.

make -j8 && make modules_install && make install

If you use Ubuntu, you can generate the kernel debian package directly and install it with below commands:

make -j8 deb-pkg

cd ..

dpkg -i linux-image-4.17.0+_4.17.0+-2_amd64.deb

3.3 Build Qemu

Please follow chapter 3.3.1 for KVMGT setup and chapter 3.3.2 for XenGT setup.

3.3.1 Build Qemu for KVMGT

git clone https://github.com/intel/igvtg-qemu

cd igvtg-qemu

git checkout stable-2.12.0

git submodule update --init roms/seabios

./configure --prefix=/usr \

    --enable-kvm \

    --disable-xen \

    --enable-libusb \

    --enable-debug-info \

    --enable-debug \

    --enable-sdl \

    --enable-vhost-net \

    --enable-spice \

    --disable-debug-tcg \

    --enable-opengl \

    --enable-gtk \

    --target-list=x86_64-softmmu

make -j8

cd roms/seabios

make -j8

cd -

make install

cp roms/seabios/out/bios.bin /usr/bin/bios.bin

3.3.2 Build Qemu & Xen for XenGT

git clone https://github.com/intel/igvtg-xen

cd igvtg-xen

git checkout xengt-stable-4.10

git clone https://github.com/intel/igvtg-qemu

cd igvtg-qemu

git checkout stable-2.12.0

cd ..

sed -i 's/\(QEMU_UPSTREAM_REVISION ?= \).*/\1origin\/stable-2.10.0/g' Config.mk

sed -i 's/\(QEMU_UPSTREAM_URL ?= \).*/\1file:\/\/localhost\/root\/igvtg-xen\/igvtg-qemu/g' Config.mk

./autogen.sh

./configure --prefix=/usr

make -j8 xen tools

make install-tools

cp xen/xen.gz /boot/xen-gvt.gz

3.4 Grub Update

You need to add the KVMGT and XenGT corrsponing menuentry in your grub.cfg, or you can add it to /etc/default/grub is better. (Since grub.cfg will be auto regenerated by linux distro version upgrade or kernel package update).

For KVMGT, Make sure the "i915.enable_gvt=1" exist in your menuentry, below is an example on ubuntu:

menuentry 'GVT-g Upstream' --class kvmgt --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
    recordfail
    load_video
    gfxmode $linux_gfx_mode
    insmod gzio
    insmod part_msdos
    insmod ext2
    set root='hd0,msdos1'
    if [ x$feature_platform_search_hint = xy ]; then
        search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  106518cd-fb74-4048-81de-3ecee7ad74d4
    else
        search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
    fi
        linux   /boot/vmlinuz-4.17+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel log_buf_len=128M console=ttyS0,115200,8n1 i915.enable_gvt=1 kvm.ignore_msrs=1 intel_iommu=on drm.debug=0
        initrd  /boot/initrd.img-4.17+
}

NOTICE: Since some Windows guest 3rd party application / tools (like GPU-Z / Passmark9.0) will trigger MSR read / write directly, if it access the unhanded msr register, guest will trigger BSOD soon. So we added the "kvm.ignore_msrs=1" into grub for workaround.

For XenGT, make sure the "i915.enable_gvt=1" exist in your menuentry, below is an example on ubuntu:

menuentry 'GVT-g Upstream' --class xengt --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
	recordfail
	load_video
	gfxmode $linux_gfx_mode
	insmod gzio
	insmod part_msdos
	insmod ext2
	set root='hd0,msdos1'
	if [ x$feature_platform_search_hint = xy ]; then
		search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  106518cd-fb74-4048-81de-3ecee7ad74d4
	else
		search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
	fi
	multiboot   /boot/xen-gvt.gz dom0_max_vcpus=2 dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all msi=1 conring_size=4M  console=com1 com1=115200,8n1 sync_console
	module   /boot/vmlinuz-4.17+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel intel_iommu=on log_buf_len=128M console=hvc0 drm.debug=0 i915.enable_gvt=1
	module  /boot/initrd.img-4.17+
}

4 Guest Setup

4.1 Linux Guest Setup

4.1.1 Linux Kernel Support

You can prepare any version Ubuntu or any other Linux guest OS image manually. Then you can change the guest kernel to the same kernel as host, you also can download the stable version from https://www.kernel.org/ or use Ubuntu18.04 guest OS kernel directly.

4.1.2 Linux Intel GFX Card Support

  • We need to make the special X11 configuration change for Linux guest to use the Intel graphics card as a display card .

  • The emulated graphics card (linke "QXL", "Cirrus")not support OpenCL / OpenGL hardware acceleration, we must make this change.

  • Here we use Ubuntu as example, we save the following section at /etc/X11/xorg.conf file:

      Section "Device"
              Identifier  "intel"
              Driver      "modesetting"
              BusID       "PCI:0:4:0"  # Sample: "PCI:0:2:0"
      EndSection
      Section "Screen"
              Identifier  "intel"
              Device "intel"
      EndSection
    

BTW: The PCI ID was chosen by looking at the output of the "lspci" command.

root@gvtg-host:~# lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]

00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]

00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)

00:02.0 VGA compatible controller: Cirrus Logic GD 5446

00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)

00:04.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 0a)

NOTICE: After you change the display card to Intel graphics card, Qemu will comes to black screen, you only can see guest desktop by remote protocal.

4.1.3 Linux Remote Protocol Support

We suggest to use the X11VNC as the remote protocol to validation which has been validated by us.

Use Ubuntu as example, to support X11VNC, you must install the xhost in guest first, the xhost program is used to add and delete host names or user names to the list allowed to make connections to the X server, you must make some configuration for lightdm or gdm.

For gdm you just need to prepare a script in profile.d for xhost, as below:

root@gvtg-host:~# cat /etc/profile.d/xhost_display.sh
#!/bin/bash
    export DISPLAY=:0

    if [ ! -z "$DISPLAY" ]; then
        xhost +local:
    fi

root@gvtg-host:~# chmod +x /etc/profile.d/xhost_display.sh

For lightdm, you also need to add the xhost configuration in lightdm configuration, as below:

root@gvtg-host:~# cat /etc/lightdm/lightdm.conf
    [SeatDefaults]
    greeter-session=unity-greeter
    user-session=ubuntu
    greeter-setup-script=/etc/profile.d/xhost_display.sh
    session-setup-script=/etc/profile.d/xhost_display.sh

After you complete the lightdm or gdm configuration, you can restart guest and run the X11VNC service in guest by following commands or similar commands:

x11vnc -display :0 -forever -bg -repeat -nowf -o ~/.vnc/x11vnc.log -rfbport 5900

Then connect to the guest by host with guest X support by typing the following command:

vncviewer $guestIP

NOTICE: $guestIP means guest IP address, you can got the guest IP address by "nmap" / "arp-scan" or any other network related tools in host.

Except X11VNC, there are some other alternatives, which were not validated by us, you also can use it as remote protocol for a try:

4.2 Windows Guest Setup

4.2.1 Windows Version Support

Win7-32 / Win7-64 / Win8.1-64 /Win10-RS4-64 are validated.

4.2.2 Windows GFX Driver Support

  • After you boot up Windows guest, you can see two graphics card in "Device manager"

  • The virtual GFX card "QXL" or "Cirrus"

  • The Intel GFX card

  • Then you must install the Intel GFX driver for the Intel Graphics card to support OpenCL / OpenGL / DirectX9 / DirectX10/ DirectX11 / DirectX12 hardware acceleration .

  • You can get the Windows driver from graphics drivers section of the Intel website. Depending on your Intel processor, the latest available driver versions as below:

    Kabylake & Skylake Win10-RS4-64: 25.20.100.6326 Kabylake & Skylake Win7-32 / Win7-64 / Win8.1-64: 15.45.23.4860 Broadwell: 15.40.41.5058

NOTICE: After you install Windows GFX driver, you can see guest desktop by both QEMU and remote protocol. QEMU display the emulated driver and the remote protocol display the Intel GFX driver. BTW, There is a known issue that some 3D workload cannot run with hardware acceleration while both GFX card is available, so suggest to disable the emulated GFX card in "Device Manager" after you make sure the Intel GFX card is working.

4.2.3 Windows Remote Protocol Support

We suggest to use Tightvnc as the remote protocol to validation which has been validated by us.

Then connect to the guest by host with guest X support by typing the following command:

vncviewer $guestIP

NOTICE: $guestIP means guest IP address, you can got the guest IP address by "nmap" / "arp-scan" or any other network related tools in host.

All Windows Remote protocol are supported, such as TightVNC, HP RGS, RDP.

Except Tightvnc, there are some other alternatives, which were not validated by us, you also can use it as remote protocol for a try:

5 Basic Usage

5.1 Check mdev Module (KVMGT only)

After you boot up you host to GVT-g menuentry, you make sure the "mdev_supported_types" node with differenct vgpu type existed, the example as below:

root@gvtg-host:~# ll /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/

total 0

drwxr-xr-x 5 root root 0 Apr 10 21:24 ./

drwxr-xr-x 12 root root 0 Apr 10 20:04 ../

drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V5_1/ $ This node with low_gm_size=512MB, high_gm_size=2048MB, fense=4, resolution=1920x1200

drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V5_2/ # This node with low_gm_size=256MB, high_gm_size=1024MB, fense=4, resolution=1920x1200

drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V5_4/ # This node with low_gm_size=128MB, high_gm_size=512MB, fense=4, resolution=1920x1200

drwxr-xr-x 3 root root 0 Apr 10 21:24 i915-GVTg_V5_8/ # This node with low_gm_size=64MB, high_gm_size=384MB, fense=4, resolution=1024x768

And you can check the detail description for each vgpu type,

root@gvtg-host:/sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_1# cat description

low_gm_size: 512MB

high_gm_size: 2048MB

fence: 4

resolution: 1920x1200

5.2 Generate the uuid for create VGPU (KVMGT only)

Use Ubuntu as example, you can use the third-party tool to generate uuid like "uuid", "uuidgen", if we need to create 3 VM, we must create 3 uuid for 3 VGPU, like below:

root@gvtg-host:~# uuid -n 3

a297db4a-f4c2-11e6-90f6-d3b88d6c9525

a297de6a-f4c2-11e6-90f7-cb6a86ce449f

a297deec-f4c2-11e6-90f8-c39f1ae774e5

5.3 Create VGPU (KVMGT only)

Use the uuid you generated to create VGPU, notes: if you want to boot up multi guests, you can use echo "uuid" > "/sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/i915-GVTg_V#ID_#Type/create", like below:

echo "a297db4a-f4c2-11e6-90f6-d3b88d6c9525" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V5_4/create"

echo "a297de6a-f4c2-11e6-90f7-cb6a86ce449f" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V5_4/create"

echo "a297deec-f4c2-11e6-90f8-c39f1ae774e5" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V5_4/create"

Then all the vgpu directory you created can be found at "/sys/bus/pci/devices/0000:00:02.0/"

I915-GVTg_V5_4 is a vGPU type name. Name I915-GVTg_Vx_y is used to represent different vGPU type. In the type name, ‘x’ means Intel GPU generation. V4 is Intel GPU GEN8 and V5 is Intel GPU GEN9. ‘y’ means this type of vGPU’s weight in GVT scheduling. GVT uses weight based round robin schedule policy. vGPU with higher weight value will get more physical GPU time slices. ‘y’ also indicates the resource assigned to this type of vGPU. Here is the mapping of ‘y’ and assigned GPU resource.

y	aperture memory size		hidden memory size		fence size

-------------------------------------------------------------------------------------------------------------

8	64				384				4

4	128				512				4

2	256				1024				4

1	512				2048				4

5.4 Remove VGPU (KVMGT only)

If you want to remove one vgpu you create, you need to "echo 1 > remove" at the vgpu directory and created, the example as below:

echo 1 > /sys/bus/pci/devices/0000:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525/remove

If you want remove all vgpu you created, you can run below command.

for i in /sys/bus/pci/devices/0000:00:02.0/*/remove; do echo 1 > $i; done

5.5 Network Bridge

For KVMGT, in order to let guest using the same network segment as host, you must create the network bridge. Besides, in order to let Qemu using the network bridge you created, you also need create a scripte as below (We saved it at /etc/qemu-ifup):

Detail you can reference the the "Configuring Guest Networking" wiki page http://www.linux-kvm.org/page/Networking

For XenGT you can just create the network bridge by yourself.

5.6 Create VM

5.6.1 Create KVMGT VM

We sugguest to use the secondary gfx card mode (virtual "qxl" or "cirrus" gfx card as 1st guest gfx card, intel gfx card as 2nd guest gfx card)to create VM, the example of create VM script as below :

#! /bin/bash -x
    /usr/bin/qemu-system-x86_64 \
    -m 2048 -smp 2 -M pc \
    -name gvt-g-guest \
    -hda /home/img/ubuntu-1.qcow2  \
    -bios /usr/bin/bios.bin -enable-kvm \
    -net nic,macaddr=00:A1:00:00:00:1A -net tap,script=/etc/qemu-ifup \
    -vga qxl \
    -k en-us \
    -serial stdio \
    -vnc :1 \
    -machine kernel_irqchip=on \
    -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \
    -cpu host -usb -usbdevice tablet \
    -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525,rombar=0,display=off,x-igd-opregion=on # for Qemu 2.12 you should add "display=off" option when you create VM without dma-buf.  

5.6.2 Create XenGT VM

You can create hvm configuration file by modifying /etc/xen/xlexample.hvm like below:

builder='hvm'
vgt=1
vcpus=2
memory = 4096
name = "win10-1"
vif = [ 'type=ioemu,bridge=xenbr0,mac=00:25:10:03:0A:01,model=e1000' ]
disk = ['/home/img/win10-red-stone.img,raw,hda,w']
device_model_version = 'qemu-xen'
device_model_override = '/usr/lib/xen/bin/qemu-system-i386'
boot="dc"
opengl=0
serial='pty'
usb=1
usbdevice='tablet'
sdl=0
vnc=1
keymap='en-us'
vgt_low_gm_sz=128
acpi_s4=0
acpi_s3=0
viridian=["all", "!apic_assist"]

NOTICE: The "vgt_low_gm_sz" is used to set vm aperture size, the supported valid value are 64 / 128 / 256 / 512 which means vm aperture size is 64MB / 128MB / 256MB / 512MB. Other invalid value will cause VM start failure.

After that, you can create XenGT VM via the below command:

xl create xlexample.hvm

6 Features Supported

6.1 Highlights Features

  • General:
    • Supported Linux distributions (64bit): Ubuntu, Fedora, RHEL, SUSE, etc
    • Supported Windows guest: Windows7 (32bit, 64bit), Windows8.1 (64bit), Windows10 RedStone (64bit)
    • Supported total vGPU number <= 7
    • Single display with resolution up to 1920x1080p
  • 3D/2D:
    • OpenGL4.5 for Linux guest
    • OpenGL4.4 for Windows guest
    • DirectX9, 10, 11, 12 for Windows guest
  • Compute:
    • OpenCL2.0 for Windows/Linux guest
  • Media:
    • Intel Media Service Studio (2015R5)
  • Remote display protocol:
    • X11VNC for Linux guest
    • TightVNC, HP RGS, RDP for Windows guest

6.2 Validated Benchmarks

  • 3D / 2D:
    • 3Dmark06, 3Dmark11, WebGL, Passmark, Unigine-tropics, Unigine-heaven, Phoronix 3D, Cairo 2D, etc
  • Media:
    • Multiple thread decode/transcode for JPEG, MPEG2, H264, HEVC, VC1, VP8, etc
  • OpenCL:
    • LuxMark, Beignet
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.