AMD GPU passthrough support

[gardenali]

It's great to have intel and nvidia support, but I'm missing the AMD option.

Thank you!

[lks-hrsch]

I am currently also at the point that I need amd gpu support to move my last service to jailmaker. Can someone explain what is needed to enable gpu support or what you have done for intel and nvidia? I am open to implementing and evaluating the feature on my system.

[easyfab]

Does it work if you manually add --bind=/dev/dri ?

e.g : jlmkr create myjail --bind=/dev/dri

[jeefberkey]

The intel gpu setting does just that

[Jip-Hop]

Jailmaker has intel and nvidia GPU support because these drivers are provided by the TrueNAS SCALE host OS. I think adding support for a dedicated AMD GPU in jailmaker is not trivial, if possible at all without modifying the host OS. Since I have no dedicated GPU in my TrueNAS server I can't investigate this. Feel free to investigate though. @lks-hrsch you could have a look at the python code of jlmkr.py to see what the intel and nvidia GPU passthrough options do.

[easyfab]

Isn't AMD support in Truenas Host OS ?

lspci -k | grep amdgpu
Kernel driver in use: amdgpu
Kernel modules: amdgpu

For info, I tried with 5700G APU, adding --bind=/dev/dri seems to give me access to the igpu in jailmaker.
Don't know if it work with dgpu.

edit : to complete

root@myjail:~# vainfo
error: can't connect to X server!
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (renoir, LLVM 15.0.6, DRM 3.54, 6.6.16-production+truenas)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

[Jip-Hop]

Yes I think there's a difference between AMD iGPU and dGPU but I'd be happy to be proven wrong.

[Jip-Hop]

I've labeled this issue as invalid and help wanted. I think iGPU is already supported (Intel or AMD). You'd have to set gpu_passthrough_intel=1 in your config file for that. I realize now this naming is confusing in this case...

Regarding AMD dedicated GPUs, as far as I know those aren't supported on the SCALE host system and therefore jailmaker can't support them either.

Since I don't have an AMD GPU I could use help to confirm this issue is indeed invalid. Either way, I won't be working on a solution for this issue and I recommend to either switch to an nvidia GPU or implement a solution (if possible) and provide a pull request.

[lks-hrsch]

I apologize for not getting back to you sooner, but I can prove that for AMD iGPU it's already working, a dGPU I also currently don't have for testing.

[maeehart]

Hey! I just want to add that that passing the AMD gpu does work, but one needs to also bind /dev/kfd. That is, the command like
./jlmkr.py create --distro=ubuntu --release=jammy ubuntu --bind=/dev/dri --bind=/dev/kfd
works. After this, I just had a bit of problems with permissions on these files. However, now I can run llama3 in a jail using an AMD GPU.

[Jip-Hop]

@maeehart is that a dedicated AMD GPU you're using (which model)? If so then that's good news and we can close this ticket as completed.

[maeehart]

Yes, it is a 6900 xt, i.e., a dedicated AMD GPU. We could still make a PR regarding the AMD support so that one could just ues the GPU by adding a gpu_passthrough_amd flag.

[Jip-Hop]

Ah yes that's a good idea. Could you provide the PR?

[maeehart]

I can do it during the weekend. I will need to see if I can do something about the permissions.

[dalgibbard]

Though you didn't specify what your permission issues are; is it fixed if you add:
--property=DeviceAllow="/dev/kfd rw"
?

I ask, since I do something similar for CoralTPU passthrough, which looks like:

--bind='/dev/ttyUSB0'
--property=DeviceAllow="/dev/ttyUSB0 rwm"
--property=DeviceAllow="char-drm rwm"
--property=DeviceAllow=/dev/bus/usb

Though I haven't spent enough time in the land of nspawn to really work out if all of these are necessary/correct lol

Edit: This made me want to go look up what "rwm" is vs just "rw", and the "m" means:

"m" (Mknod): Allows the creation of device nodes using mknod. Device nodes are special files in Unix-like operating systems that represent device interfaces. With this permission, the container can create new device nodes within its filesystem, enabling access to devices that were not initially available. This is useful for dynamically creating device nodes as needed by containerized applications.

[Jip-Hop]

Please have a look in the jlmkr.py code and search for for DeviceAllow. I think adding this explicitly will actually cause issues instead of solving them.

[jere-co]

Any updates on the AMD dGPU support?