The research prototype of GPUvm.
We tested & evaluated on the following environments.
- gxen stable branch OR latest
- Linux 3.6.5 kernel
- NVIDIA Quadro6000 GPU
- We tested with NVIDIA Quadro6000 nvc0 GPU
- Kernel mode gdev
605e69e70ce7b4c505be91696612e98649ec383fis tested (In this version, cubin should be built with NVCC 4.2. This is caused by this kernel mode gdev's limitation)
- But reverting
d4a6697583ca3d5606711402be121da0bf9875e2is recommended since it acquires a lot of memory (since we only support BAR remapping for BAR3 (not BAR1: it is only used for submitting a command request on the NVIDIA driver. Submission requests are trapped by gxen) in this prototype, initializing this memory area takes a lot of time)
- But reverting
- Rodinia benchmarks
Build Xen part
stubdom cannot be built since we use C++ in qemu-dm nvc0 device model. So in order to build it, try
make xen make tools make install-xen make install-tools
And boost, g++, libpciaccess libraries are necessary.
Build A3 (GPU Access Aggregator) part
A3 is located on
tools/a3. To build it (stable branch), try
make -C tools/a3/build
To build it (in latest branch), try
cd tools/a3 cmake -H. -Bout make -C out
Then you get a3 binary on
tools/a3/out/a3. You can see options by providing
usage: a3 [options] [program_file] [arguments] options: -h, --help print this message -v, --version print the version -t, --through through I/O --lazy-shadowing Enable lazy shadowing --bar3-remapping Enable BAR3 remapping
To follow the old gdev kernel module build instruction. Generate gdev.ko from that.
Build Linux kernel
Get Linux kernel 3.6.5 from the above tree and build it.
Initialization part depends on the host nouveau driver. So you need to load the 3.6.5 nouveau on host environment. At this time, you need to specify the following kernel command line to Host Linux on Xen,
And you don't use X-server since it may load nouvea & use it. You need to load host nouveau driver, but it is used for initializing GPU and you must not touch GPU from the host.
And don't enable Xen iommu support, gxen doesn't use VT-d.
To expose virtualized GPU, writing
nvc0=0 to your Xen's hvm file.
Need to execute
a3 with an appropriate GPU device bdf number.
Boot Xen HVM with above Linux 3.6.5 kernel
Boot Xen HVM with a virtualized GPU.
In this time, you need to specify the following command to VM's Linux commandline to make gdev works,
Before booting HVM, you need to run
a3 command with bus number. such as
build/a3 --bar3-remapping --lazy-shadowing 0300
Recommend not to load X-server on HVM.
Load gdev module on HVM
And then, you need to load gdev.ko. Follow the gdev kernel module instructions.
Run GPU benchmarks
You can run GPU benchmarks with gxen. For example,
You can load multiple VMs that use virtualized GPU.
But the resources (such as channels, device memory) are splitted statically, so you need to adjust
And if you change this configuration, remember to build Xen tools by executing
make tools and