GVTg_Setup_Guide
- 1 Introduction
- 2 System Requirements
- 3 Host Environment Setup
- 4 Guest Setup
- 5 Basic Usage
- 6 Features Supported
Intel GVT-g is a full GPU virtualization solution with mediated pass-through (VFIO mediated device framework based), starting from 5th generation Intel Core(TM) processors with Intel Graphics processors. GVT-g supports both Xen and KVM (a.k.a XenGT & a.k.a KVMGT). A virtual GPU instance is maintained for each VM, with part of performance critical resources directly assigned. The capability of running native graphics driver inside a VM, without hypervisor intervention in performance critical paths, achieves a good balance among performance, feature, and sharing capability.
Ubuntu 20.04.3 has been fully validated as host, other Linux operating system like RHEL / Fedora are also OK.
For client platforms, 5th, 6th, 7th, 8th or 7th SoC Generation Intel® Core Processor Graphics is required.
For server platforms, E3_v4, E3_v5 or E3_v6 Xeon Processor Graphics is required.
Use Ubuntu as an example, there are some software package needed by host environment setup, as below:
apt-get install git libfdt-dev libpixman-1-dev libssl-dev vim socat libsdl1.2-dev libspice-server-dev autoconf libtool xtightvncviewer tightvncserver x11vnc libsdl1.2-dev uuid-runtime uuid uml-utilities bridge-utils python-dev liblzma-dev libc6-dev libegl1-mesa-dev libepoxy-dev libdrm-dev libgbm-dev libaio-dev libusb-1.0.0-dev libgtk-3-dev bison
GVT-g kernel: https://github.com/intel/gvt-linux/ (branch: gvt-staging or gvt-stable-4.17)
QEMU: https://github.com/intel/igvtg-qemu (branch: stable-2.12.0)
Xen : https://github.com/intel/igvtg-xen (branch: xengt-stable-4.10)
NOTICE: For KVMGT, you also can use the current upstream Linux kernel (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/) and QEMU (https://git.qemu.org/git/qemu.git) directly since all the enabling patches have been upstreamed.
For XenGT, you must use the repositories we provided and it will not be updated anymore.
Use Ubuntu as example, modify /etc/initramfs-tools/modules, like below:
kvmgt
xengt
vfio-iommu-type1
vfio-mdev
For Fedora or RHEL you should make the similar changes in "dracut".
git clone https://github.com/intel/gvt-linux.git
cd gvt-linux
git checkout gvt-stable-4.17
echo ""|make oldconfig
Then make sure to enable CONFIG_DRM_I915_GVT, CONFIG_DRM_I915_GVT_KVMGT and CONFIG_DRM_I915_GVT_XENGT in ".config", which depends on CONFIG_VFIO_MDEV and CONFIG_VFIO_MDEV_DEVICE.
make -j8 && make modules_install && make install
If you use Ubuntu, you can generate the kernel debian package directly and install it with below commands:
make -j8 deb-pkg
cd ..
dpkg -i linux-image-4.17.0+_4.17.0+-2_amd64.deb
Please follow chapter 3.3.1 for KVMGT setup and chapter 3.3.2 for XenGT setup.
git clone https://github.com/intel/igvtg-qemu
cd igvtg-qemu
git checkout stable-2.12.0
git submodule update --init roms/seabios
./configure --prefix=/usr \
--enable-kvm \
--disable-xen \
--enable-libusb \
--enable-debug-info \
--enable-debug \
--enable-sdl \
--enable-vhost-net \
--enable-spice \
--disable-debug-tcg \
--enable-opengl \
--enable-gtk \
--target-list=x86_64-softmmu
make -j8
cd roms/seabios
make -j8
cd -
make install
cp roms/seabios/out/bios.bin /usr/bin/bios.bin
git clone https://github.com/intel/igvtg-xen
cd igvtg-xen
git checkout xengt-stable-4.10
git clone https://github.com/intel/igvtg-qemu
cd igvtg-qemu
git checkout stable-2.12.0
cd ..
sed -i 's/\(QEMU_UPSTREAM_REVISION ?= \).*/\1origin\/stable-2.12.0/g' Config.mk
sed -i 's/\(QEMU_UPSTREAM_URL ?= \).*/\1file:\/\/localhost\/root\/igvtg-xen\/igvtg-qemu/g' Config.mk
./autogen.sh
./configure --prefix=/usr
make -j8 xen tools
make install-tools
cp xen/xen.gz /boot/xen-gvt.gz
You need to add the KVMGT and XenGT corrsponing menuentry in your grub.cfg, or you can add it to /etc/default/grub is better. (Since grub.cfg will be auto regenerated by linux distro version upgrade or kernel package update).
For KVMGT, Make sure the "i915.enable_gvt=1" exist in your menuentry, below is an example on ubuntu:
menuentry 'GVT-g Upstream' --class kvmgt --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 106518cd-fb74-4048-81de-3ecee7ad74d4
else
search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
fi
linux /boot/vmlinuz-4.17+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel log_buf_len=128M console=ttyS0,115200,8n1 i915.enable_gvt=1 kvm.ignore_msrs=1 intel_iommu=igfx_off drm.debug=0
initrd /boot/initrd.img-4.17+
}
NOTICE: Since some Windows guest 3rd party application / tools (like GPU-Z / Passmark9.0) will trigger MSR read / write directly, if it access the unhanded msr register, guest will trigger BSOD soon. So we added the "kvm.ignore_msrs=1" into grub for workaround.
For XenGT, make sure the "i915.enable_gvt=1" exist in your menuentry, below is an example on ubuntu:
menuentry 'GVT-g Upstream' --class xengt --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-106518cd-fb74-4048-81de-3ecee7ad74d4' {
recordfail
load_video
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 106518cd-fb74-4048-81de-3ecee7ad74d4
else
search --no-floppy --fs-uuid --set=root 106518cd-fb74-4048-81de-3ecee7ad74d4
fi
multiboot /boot/xen-gvt.gz dom0_max_vcpus=2 dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all msi=1 conring_size=4M console=com1 com1=115200,8n1 sync_console
module /boot/vmlinuz-4.17+ root=UUID=106518cd-fb74-4048-81de-3ecee7ad74d4 ro ignore_loglevel intel_iommu=on log_buf_len=128M console=hvc0 drm.debug=0 i915.enable_gvt=1
module /boot/initrd.img-4.17+
}
You can prepare any version Ubuntu or any other Linux guest OS image manually. Then you can change the guest kernel to the same kernel as host, you also can download the stable version from https://www.kernel.org/ or use Ubuntu18.04 guest OS kernel directly.
-
We need to make the special X11 configuration change for Linux guest to use the Intel graphics card as a display card .
-
The emulated graphics card (linke "QXL", "Cirrus")not support OpenCL / OpenGL hardware acceleration, we must make this change.
-
Here we use Ubuntu as example, we save the following section at /etc/X11/xorg.conf file:
Section "Device" Identifier "intel" Driver "modesetting" BusID "PCI:0:4:0" # Sample: "PCI:0:2:0" EndSection Section "Screen" Identifier "intel" Device "intel" EndSection
BTW: The PCI ID was chosen by looking at the output of the "lspci" command.
root@gvtg-host:~# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
00:04.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 0a)
NOTICE: After you change the display card to Intel graphics card, Qemu will come to black screen, you only can see guest desktop by remote protocol.
We suggest to use the X11VNC as the remote protocol to validation which has been validated by us.
Use Ubuntu as example, to support X11VNC, you must install the xhost in guest first, the xhost program is used to add and delete host names or user names to the list allowed to make connections to the X server, you must make some configuration for lightdm or gdm.
For gdm you just need to prepare a script in profile.d for xhost, as below:
root@gvtg-guest:~# cat /etc/profile.d/xhost_display.sh
#!/bin/bash
export DISPLAY=:0
if [ ! -z "$DISPLAY" ]; then
xhost +local:
fi
root@gvtg-vm:~# chmod +x /etc/profile.d/xhost_display.sh
For lightdm, you also need to add the xhost configuration in lightdm configuration, as below:
root@gvtg-guest:~# cat /etc/lightdm/lightdm.conf
[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu
greeter-setup-script=/etc/profile.d/xhost_display.sh
session-setup-script=/etc/profile.d/xhost_display.sh
After you complete the lightdm or gdm configuration, you can restart guest and run the X11VNC service in guest by following commands or similar commands:
x11vnc -display :0 -forever -bg -repeat -nowf -o ~/.vnc/x11vnc.log -rfbport 5900
Then connect to the guest by host with guest X support by typing the following command:
vncviewer $guestIP
NOTICE: $guestIP means guest IP address, you can got the guest IP address by "nmap" / "arp-scan" or any other network related tools in host.
Except X11VNC, there are some other alternatives, which were not validated by us, you also can use it as remote protocol for a try:
- NX
- X2Go and based on nx-libs (it seems QVD uses it too)
- QVD (open source version)
- NoMachine: Closed source.
- OpenNX. Client only. Not updated since 2016-10-01.
- VNC
- TurboVNC
Win7-32 / Win7-64 / Win8.1-64 /Win10-RS4-64 are validated.
-
After you boot up Windows guest, you can see two graphics card in "Device manager"
-
The virtual GFX card "QXL" or "Cirrus"
-
The Intel GFX card
-
Then you must install the Intel GFX driver for the Intel Graphics card to support OpenCL / OpenGL / DirectX9 / DirectX10/ DirectX11 / DirectX12 hardware acceleration .
-
You can get the Windows driver from graphics drivers section of the Intel website. Depending on your Intel processor, the latest available driver versions as below:
Kabylake & Skylake & Coffeelake Win10-64: 30.0.100.9667
Skylake Win7-32 / Win7-64 / Win8.1-64: 15.45.31.5127
Broadwell: 15.40.45.5126
NOTICE: After you install Windows GFX driver, you can see guest desktop by both QEMU and remote protocol. QEMU display the emulated driver and the remote protocol display the Intel GFX driver. BTW, There is a known issue that some 3D workload cannot run with hardware acceleration while both GFX card is available, so suggest to disable the emulated GFX card in "Device Manager" after you make sure the Intel GFX card is working.
We suggest to use Tightvnc as the remote protocol to validation which has been validated by us.
Then connect to the guest by host with guest X support by typing the following command:
vncviewer $guestIP
NOTICE: $guestIP means guest IP address, you can got the guest IP address by "nmap" / "arp-scan" or any other network related tools in host.
All Windows Remote protocol are supported, such as TightVNC, HP RGS, RDP.
Except Tightvnc, there are some other alternatives, which were not validated by us, you also can use it as remote protocol for a try:
- VNC (Virtual Network Computing)
- HP's RGS (Remote Graphics Software), which is an enterprise-level application.
- Microsoft's RDP (Remote Desktop Protocol).
- FreeRDP: The best Open Source RDP implementation.
- Remmina (GTK+ 3): Based on FreeRDP. It supports the RDP, VNC, NX, XDMCP and SSH protocols.
- WinConn: Promising but abandoned project. Just put here as reference, not useful for end users (the readers of this document).
- PAC Manager
- KDE's KRDC (Remote Display Client): It supports both VNC and RDP protocols.
- Gnome's Vinagre
- NX
After you boot up you host to GVT-g menuentry, you make sure the "mdev_supported_types" node with differenct vgpu type existed, the example as below:
root@gvtg-host:~# ls /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/
i915-GVTg_V5_1 i915-GVTg_V5_2 i915-GVTg_V5_4 i915-GVTg_V5_8
And you can check the description for each vgpu type, the example as below:
root@gvtg-host:/sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_1# cat description
low_gm_size: 512MB
high_gm_size: 2048MB
fence: 4
resolution: 1920x1200
weight: 16
Use Ubuntu as example, you can use the third-party tool to generate uuid like "uuid", "uuidgen", if we need to create 3 VM, we must create 3 uuid for 3 VGPU, like below:
root@gvtg-host:~# uuid -n 3
a297db4a-f4c2-11e6-90f6-d3b88d6c9525
a297de6a-f4c2-11e6-90f7-cb6a86ce449f
a297deec-f4c2-11e6-90f8-c39f1ae774e5
Use the uuid you generated to create VGPU, notes: if you want to boot up multi guests, you can use echo "uuid" > "/sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/i915-GVTg_V#ID_#Type/create", like below:
echo "a297db4a-f4c2-11e6-90f6-d3b88d6c9525" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V5_4/create"
echo "a297de6a-f4c2-11e6-90f7-cb6a86ce449f" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V5_4/create"
echo "a297deec-f4c2-11e6-90f8-c39f1ae774e5" > "/sys/bus/pci/devices/0000:00:02.0/ mdev_supported_types/i915-GVTg_V5_4/create"
Then all the vgpu directory you created can be found at "/sys/bus/pci/devices/0000:00:02.0/"
I915-GVTg_V5_4 is a vGPU type name. Name I915-GVTg_Vx_y is used to represent different vGPU type: ‘x’ indicates Intel GPU generation. That V4 is Intel GPU GEN8 and V5 is Intel GPU GEN9; ‘y’ is the type ID. The smaller the ID is, the more resource the type has. The detailed resource mapping to type is in below table. Each type has a ‘weight’ setting as well. vGPU with higher weight value will get more physical GPU time slices.
y aperture memory size hidden memory size fence size weight
-------------------------------------------------------------------------------------------------------------
8 64 384 4 2
4 128 512 4 4
2 256 1024 4 8
1 512 2048 4 16
If you want to remove one vgpu you create, you need to "echo 1 > remove" at the vgpu directory and created, the example as below:
echo 1 > /sys/bus/pci/devices/0000:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525/remove
If you want remove all vgpu you created, you can run below command.
for i in /sys/bus/pci/devices/0000:00:02.0/*/remove; do echo 1 > $i; done
For KVMGT, in order to let guest using the same network segment as host, you must create the network bridge. Besides, in order to let Qemu using the network bridge you created, you also need create a scripte as below (We saved it at /etc/qemu-ifup):
Detail you can reference the the "Configuring Guest Networking" wiki page http://www.linux-kvm.org/page/Networking
For XenGT you can just create the network bridge by yourself.
We sugguest to use the secondary gfx card mode (virtual "qxl" or "cirrus" gfx card as 1st guest gfx card, intel gfx card as 2nd guest gfx card)to create VM, the example of create VM script as below :
#! /bin/bash -x
/usr/bin/qemu-system-x86_64 \
-m 2048 -smp 2 -M pc \
-name gvt-g-guest \
-hda /home/img/ubuntu-1.qcow2 \
-bios /usr/bin/bios.bin -enable-kvm \
-net nic,macaddr=00:A1:00:00:00:1A -net tap,script=/etc/qemu-ifup \
-vga qxl \
-k en-us \
-serial stdio \
-vnc :1 \
-machine kernel_irqchip=on \
-global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \
-cpu host -usb -usbdevice tablet \
-device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/a297db4a-f4c2-11e6-90f6-d3b88d6c9525,rombar=0,display=off,x-igd-opregion=on # for Qemu 2.12 you should add "display=off" option when you create VM without dma-buf.
You can create hvm configuration file by modifying /etc/xen/xlexample.hvm like below:
builder='hvm'
vgt=1
vcpus=2
memory = 4096
name = "win10-1"
vif = [ 'type=ioemu,bridge=xenbr0,mac=00:25:10:03:0A:01,model=e1000' ]
disk = ['/home/img/win10-red-stone.img,raw,hda,w']
device_model_version = 'qemu-xen'
device_model_override = '/usr/lib/xen/bin/qemu-system-i386'
boot="dc"
opengl=0
serial='pty'
usb=1
usbdevice='tablet'
sdl=0
vnc=1
keymap='en-us'
vgt_low_gm_sz=128
acpi_s4=0
acpi_s3=0
viridian=["all", "!apic_assist"]
NOTICE: The "vgt_low_gm_sz" is used to set vm aperture size, the supported valid value are 64 / 128 / 256 / 512 which means vm aperture size is 64MB / 128MB / 256MB / 512MB. Other invalid value will cause VM start failure.
After that, you can create XenGT VM via the below command:
xl create xlexample.hvm
- General:
- Supported Linux distributions (64bit): Ubuntu, Fedora, RHEL, SUSE, etc
- Supported Windows guest: Windows7 (32bit, 64bit), Windows8.1 (64bit), Windows10 RedStone (64bit)
- Supported total vGPU number <= 7
- Single display with resolution up to 1920x1080p
- 3D/2D:
- OpenGL4.5 for Linux guest
- OpenGL4.4 for Windows guest
- DirectX9, 10, 11, 12 for Windows guest
- Compute:
- OpenCL2.0 for Windows/Linux guest
- Media:
- Intel Media Service Studio (2015R5)
- Remote display protocol:
- X11VNC for Linux guest
- TightVNC, HP RGS, RDP for Windows guest
- 3D / 2D:
- 3Dmark06, 3Dmark11, WebGL, Passmark, Unigine-tropics, Unigine-heaven, Phoronix 3D, Cairo 2D, etc
- Media:
- Multiple thread decode/transcode for JPEG, MPEG2, H264, HEVC, VC1, VP8, etc
- OpenCL:
- LuxMark, Beignet