Setup to make it work on virtual machine

[insujang]

Hi, I opened a new issue from here with more specific topic.
I'm not sure this repo is more suitable or this repo more.

I want to run AMD ROC framework on virtual machine. However, it is not possible for my environment.
Is there anyone who tried to run it and succeed on virtual machine? Could you show tell me your setup? I can use Xen or KVM, and currently trying to run it on KVM.

Here is my setup:

Host
CPU: Intel Core i7 6700 (IOMMU enabled)
GPU: AMD Radeon RX 480 8GB
OS: Ubuntu 16.04.2 LTS (kernel 4.11)
I confirmed the sample worked if I run right on the host machine with kernel 4.9.0-kfd (ROCK 1.6).

Guest
Hypervisor: QEMU(2.7.92)-KVM(kernel 4.11)
Guest OS: Ubuntu 16.04.2 LTS (kernel 4.9.0-kfd)
VGA passthrough via vfio-pci
QEMU script:

sudo vfio-bind 0000:01:00.0 0000:01:00.1 &&

sudo qemu-system-x86_64 -enable-kvm -cpu host,kvm=off \
-m 8192 -smp 8 -M q35 \
-vga none -nographic \
-usb \
-device usb-host,hostbus=1,hostaddr=2 \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd \
-drive if=pflash,format=raw,file=OVMF_VARS.fd \
-hda ubuntu.qcow2 \
-net nic \
-net user,hostfwd=tcp::54321-:22

Both guest machine and host machine freeze when I use ioh3420 PCIe controller, so I directly attach the GPU to root PCI bus.

My error message is

kfd kfd: skipped device (1002:67df), PCI rejects atomics

and it makes no GPU agent.
I am sure my hardware supports PCIe atomics, just the virtualized controller doesn't.

How can I make it work?
Any suggestion?

Thanks in advance!

[insujang]

Well, I found ROCm supports KVM passthrough since version 1.3.

http://www.amd.com/en-us/press-releases/Pages/rocm-platform-2016nov14.aspx

ROCm Virtualization of the GPU hardware via OS Containers and  
Linux®'s Kernel Virtual Machine (KVM) - ...  
ROCm also supports GPU Hardware Virtualization via KVM pass-through  
to allow the benefits of hardware-accelerated GPU computing in virtualized solutions.

Is there any special flag to enable it?

[insujang]

Solved with this fix.
ROCm/ROCK-Kernel-Driver#26