obs-ffmpeg: Make AMF encoder work on Linux

[nowrep]

Description

Add Linux support for AMF encoder.
Only the fallback encoders are available (no texture support).

Requires AMD proprietary Vulkan driver, using different driver
will be detected on startup and the encoders disabled.

Motivation and Context

AMF is also available on Linux and thus users on Linux should be able to use it in OBS too.

How Has This Been Tested?

Tested with RX 6700 XT using latest driver version 22.20 on Arch Linux.

VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/amd_pro_icd64.json obs

Types of changes

• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[GloriousEggroll]

nowrep once again making awesome things happen on linux. this plugin works great.

For Nobara (Fedora) we set this up so our users can use the AMF encoder easily:

https://github.com/CosmicFusion/fedora-amdgpu-pro

I just tested your plugin on a 6900 XT and was able to record 1440p60 just fine.
It should be noted that anything higher than 1440p results in a black video recording (this is a limitation of AMF -- it happens on ffmpeg directly and with the StreamFX plugin), although this may be a limitation only on RDNA2 due to this bug:

GPUOpen-LibrariesAndSDKs/AMF#334

I'll try to test later on the full official driver stack on ubuntu

[GloriousEggroll]

nowrep once again making awesome things happen on linux. this plugin works great.

For Nobara (Fedora) we set this up so our users can use the AMF encoder easily:

https://github.com/CosmicFusion/fedora-amdgpu-pro

I just tested your plugin on a 6900 XT and was able to record 1440p60 just fine. It should be noted that anything higher than 1440p results in a black video recording (this is a limitation of AMF -- it happens on ffmpeg directly and with the StreamFX plugin), although this may be a limitation only on RDNA2 due to this bug:

GPUOpen-LibrariesAndSDKs/AMF#334

I'll try to test later on the full official driver stack on ubuntu

UPDATE: After digging around the pro packaging, we got h.265 to work via custom ffmpeg -- it needs amdgpu's libdrm (we packaged that today in the same repo) and opencl-filesystem from fedora repos -- these two fix the GPUOpen bug listed (ill post an update letting them know). If we can get the h.265 option added to this PR i'd be happy to test.

[nowrep]

It checks for hevc support at runtime, you can test it manually by running obs-amf-test.
It doesn't work for me, presumably because of that GPUOpen issue and having to use old vulkan driver - so no features added recently (hevc support, B-frames, ...) work on my AMF installation.

[GloriousEggroll]

It checks for hevc support at runtime, you can test it manually by running obs-amf-test. It doesn't work for me, presumably because of that GPUOpen issue and having to use old vulkan driver - so no features added recently (hevc support, B-frames, ...).

it's working here with custom ffmpeg with the options i noted -- do you have discord or something we can have a discussion about it? if you're on fedora i can help get you sorted on the amd pro driver side

[nowrep]

I prefer e-mail or IRC nowrep at libera.chat

[nowrep]

Changes:

• Try to load AMF library before running obs-amf-test
• Fixed HEVC detection

[GloriousEggroll]

Just dropping in a confirmation that this works great now, worked with nowrep to get the drivers sorted on his side and everything's all good:
https://www.youtube.com/watch?v=jlSEf_TPfPA

[CosmicFusion]

i have perfected my amdgpu-pro repackging (what @GloriousEggroll uses in nobara) , and this plugin works well on all gpus , with 4k support on all gpus (not great on lower end ones like polaris) , and h265 on RX5XXX and RX6XXX ..

[nowrep]

You only need to install latest amf and amdgpu-pro vulkan driver, nothing else. For Arch see Ashark/archlinux-amdgpu-pro#78
No libdrm or anything is needed, what made the drivers work on Arch was the amdgpu firmware update.

Also please keep the driver install troubleshooting outside this pull request. Pull request is ready for review, everything works just the same as on Windows (assuming your drivers are installed correctly, and of course without texture encoding).

[p0ryae]

Just tested this PR & cloned the following repo and installed it: https://github.com/HannesMann/archlinux-amdgpu-pro

I can safely say that everything is functional when using the right amd-pro driver.
This pr makes the AMF encoders viable in the encoder option, if the pro driver is installed properly and has the v22.20.3 or higher. No bugs experienced at all at the current state of this PR.

Tested on Arch Linux & an RX 6700 XT

[niobium93]

Forgive my ignorance, but is there any benefit to using this instead of something like this GStreamer based obs-vaapi or the older obs-gstreamer plugins or the even older hevc-vaapi patch? I've used all of those for HEVC encoding on my RX 5700 XT and they've all worked fine.

[alosarjos]

Forgive my ignorance, but is there any benefit to using this instead of something like this GStreamer based obs-vaapi or the older obs-gstreamer plugins or the even older hevc-vaapi patch? I've used all of those for HEVC encoding on my RX 5700 XT and they've all worked fine.

I get a huge quality difference with way lower bitrate. Specially useful if streaming rather than recording

[mesmerx]

@DashCruft i use this same spec but the https://github.com/HannesMann/archlinux-amdgpu-pro dont work for me,
with the -fix packages from aur i get the h264 to work but not hevc, can you say the packages or what you did for this work for you?

[GloriousEggroll]

latest amd firmware from 2022-09-13 (on fedora its amd-gpu-firmware, others may be linux-firmware) broke AMF again. Last known working firmware was 2022-08-15. If you dowgrade firmware packages please note you also have to regenerate initramfs and reboot.

On fedora 36:
sudo dnf downgrade amd-gpu-firmware
sudo dracut -f
sudo reboot

Please note this is a firmware issue and -not- an issue with this PR.

[mesmerx]

@GloriousEggroll oh and I get something strange in my obs, I don't know if it's because of all this strange setup that is needed for the AMF to work.
But when I stream to twitch with the AMF in the obs everything seems to be fine but the live never goes on.
If I stream to restream the image and sound appears in restream dashboard,but even this when restream to twitch never goes online.
I don't know exactly how to debug this.
I'll try to make the AMF to work in the last version before return here with more information, but I thing it's good to alert this happens to me

[mesmerx]

Just to confirm with the firmware 08-15 and the https://github.com/HannesMann/archlinux-amdgpu-pro now my hevc seem to work and the twitch bug are gone, great work ppl :)

arch with 6700xt too

[Big-FG]

On my new RX 6800 this is literally the only way to get decent encoding quality. obs-vaapi/gstreamer-vaapi or the built in vaapi either require bit rate so high you can only record or the quality is so bad its unusable. My old RX480 is okish with VAAPI but i guess not RDNA2.

Works flawlessly and the quality is the best you can get with an AMDGPU, only better option on linux for AMDGPUs would be when the bargain bin ARC GPUs come rolling out if they have an encoder as a dedicated card.

Currently on Arch you need just the amf-amdgpu-pro and the linux-firmware 20220815 like has been mentioned but otherwise easy to setup.

[nowrep]

Again I'll ask to keep troubleshooting out of this PR, thanks.

@daedalao Your build is missing jack plugin.

[Frozen-byte]

This build sadly did not work for me. If the issue is on my side, feel free to contact me helping me to fix the issue, since @nowrep asked us to keep the troubleshooting out of this.

Using Kubuntu 22.04.01
GPU is a 6900XT
installed amf command amdgpu-install --usecase=graphics,multimedia,amf --vulkan=amdvlk,pro

checked out obs/master and fetch-cherry-picked commit e2a95d6
built obs with CI/build-linux.sh

amf Test was positive

    frozen_byte@frozen-pc:~/Projects/obs-studio$ build/rundir/RelWithDebInfo/bin/obs-amf-test 
    0122-10-01 01:28:00 8123C740 [AMFEncoderCoreBaseImpl] Warning: Failed to QueryThroughput()!
    0122-10-01 01:28:00 8123C740 [AMFEncoderCoreBaseImpl] Warning: Failed to QueryThroughput()!
    [0]
    is_amd=true
    supports_avc=true
    supports_hevc=true

Here is my startscript with the relevant env vars

cd into /usr is necessary because I had some issues with loading the locale

!#/usr/bin/bash
cd /usr; LD_LIBRARY_PATH=/opt/amdgpu-pro/lib/x86_64-linux-gnu:/opt/amdgpu-pro/lib/i386-linux-gnu:"$LD_LIBRARY_PATH" VK_ICD_FILENAMES=/opt/amdgpu-pro/etc/vulkan/icd.d/amd_icd64.json:/opt/amdgpu-pro/etc/vulkan/icd.d/amd_icd32.json OBS_USE_EGL=1 /home/frozen_byte/Projects/obs-studio/build/rundir/RelWithDebInfo/bin/obs #AMFOBS

And finally the obs output

frozen_byte@frozen-pc:~/Projects/obs-studio$ ./start-build.sh

./start-build.sh: line 1: !#/usr/bin/bash: No such file or directory

debug: Found portal inhibitor

Attempted path: share/obs/obs-studio/locale/en-US.ini

Attempted path: share/obs/obs-studio/locale.ini

Attempted path: share/obs/obs-studio/themes/System.qss

info: Using EGL/X11

info: CPU Name: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz

info: CPU Speed: 1312.775MHz

info: Physical Cores: 4, Logical Cores: 4

info: Physical Memory: 15939MB Total, 9441MB Free

info: Kernel Version: Linux 5.15.0-52-generic

info: Distribution: "Ubuntu" "22.04"

info: Session Type: x11

info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.21.1

info: Qt Version: 6.2.4 (runtime), 6.2.4 (compiled)

info: Portable mode: false

Attempted path: share/obs/obs-studio/themes/Dark/no_sources.svg

qt.core.qmetaobject.connectslotsbyname: QMetaObject::connectSlotsByName: No matching signal for on_tbar_position_valueChanged(int)

qt.core.qmetaobject.connectslotsbyname: QMetaObject::connectSlotsByName: No matching signal for on_actionShowTransitionProperties_triggered()

qt.core.qmetaobject.connectslotsbyname: QMetaObject::connectSlotsByName: No matching signal for on_actionHideTransitionProperties_triggered()

info: OBS 28.1.0-beta1-54-g1fc6c50b6-modified (linux)

info: ---------------------------------

info: ---------------------------------

info: audio settings reset:

samples per sec: 48000

speakers:        2

max buffering:   960 milliseconds

buffering type:  dynamically increasing

info: ---------------------------------

info: Initializing OpenGL...

info: Loading up OpenGL on adapter AMD AMD Radeon RX 6900 XT (sienna_cichlid, LLVM 14.0.1, DRM 3.46, 5.15.0-52-generic)

info: OpenGL loaded successfully, version 4.6 (Core Profile) Mesa 22.1.0-devel, shading language 4.60

info: ---------------------------------

info: video settings reset:

base resolution:   1920x1080

output resolution: 1920x1080

downscale filter:  Bicubic

fps:               60/1

format:            NV12

YUV mode:          Rec. 709/Full

info: NV12 texture support not available

info: P010 texture support not available

info: Audio monitoring device:

name: Default

id: default

info: ---------------------------------

warning: No AJA devices found, skipping loading AJA UI plugin

warning: Failed to initialize module 'aja-output-ui.so'

warning: No AJA devices found, skipping loading AJA plugin

warning: Failed to initialize module 'aja.so'

warning: Failed to load 'en-US' text for module: 'decklink-captions.so'

warning: Failed to load 'en-US' text for module: 'decklink-output-ui.so'

libDeckLinkAPI.so: cannot open shared object file: No such file or directory

warning: A DeckLink iterator could not be created.  The DeckLink drivers may not be installed

warning: Failed to initialize module 'decklink.so'

info: [pipewire] Available captures:

info: [pipewire]     - Desktop capture

info: [pipewire]     - Window capture

info: [linux-vkcapture] plugin loaded successfully (version 1.2.1)

info: [obs-browser]: Version 2.18.5

info: [obs-browser]: CEF Version 103.0.5060.134 (runtime), 103.0.0-5060-shared-textures_143.2591+g4204d54+chromium-103.0.5060.134 (compiled)

info: FFMPEG VAAPI supported

info: [obs-websocket] [obs_module_load] you can haz websockets (Version: 5.0.1 | RPC Version: 1)

info: [obs-websocket] [obs_module_load] Qt version (compile-time): 6.2.4 | Qt version (run-time): 6.2.4

info: [obs-websocket] [obs_module_load] Linked ASIO Version: 101201

info: [obs-websocket] [obs_module_load] Module loaded.

info: [vlc-video]: VLC 3.0.16 Vetinari found, VLC video source enabled

info: ---------------------------------

info:   Loaded Modules:

info:     linux-pipewire-audio.so

info:     vlc-video.so

info:     text-freetype2.so

info:     rtmp-services.so

info:     obs-x264.so

info:     obs-websocket.so

info:     obs-vst.so

info:     obs-transitions.so

info:     obs-outputs.so

info:     obs-libfdk.so

info:     obs-filters.so

info:     obs-ffmpeg.so

info:     obs-browser.so

info:     linux-vkcapture.so

info:     linux-v4l2.so

info:     linux-pulseaudio.so

info:     linux-pipewire.so

info:     linux-jack.so

info:     linux-capture.so

info:     linux-alsa.so

info:     image-source.so

info:     frontend-tools.so

info:     decklink-output-ui.so

info:     decklink-captions.so

info: ---------------------------------

QLayout: Attempting to add QLayout "" to OBSBasic "OBSBasic", which already has a layout

QWidget::setTabOrder: 'first' and 'second' must be in the same window

info: ==== Startup complete ===============================================

[mesmerx]

This build sadly did not work for me. If the issue is on my side, feel free to contact me helping me to fix the issue, since @nowrep asked us to keep the troubleshooting out of this.

Using Kubuntu 22.04.01 GPU is a 6900XT installed amf command amdgpu-install --usecase=graphics,multimedia,amf --vulkan=amdvlk,pro

checked out obs/master and fetch-cherry-picked commit e2a95d6 built obs with CI/build-linux.sh
amf Test was positive
Here is my startscript with the relevant env vars
cd into /usr is necessary because I had some issues with loading the locale
And finally the obs output

can you try a make like this https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=obs-studio-amf?

[CosmicFusion]

This merge request seems to only work windows, those encoders never pop up on Linux no matter what, this pr is to get those encoders on Linux.

…

________________________________ From: Lucas de Paula Ferreira ***@***.***> Sent: Saturday, November 19, 2022 6:15:16 AM To: obsproject/obs-studio ***@***.***> Cc: Ward Nakchbandi (Cosmic Fusion) ***@***.***>; Comment ***@***.***> Subject: Re: [obsproject/obs-studio] obs-ffmpeg: Make AMF encoder work on Linux (PR #7206) #6508<#6508> seems to be merged and adds this func too — Reply to this email directly, view it on GitHub<#7206 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AT63FLONA2JT6EYSZAYXWIDWJBH5JANCNFSM572VZKJA>. You are receiving this because you commented.Message ID: ***@***.***>

[Fenrirthviti]

This merge request seems to only work windows, those encoders never pop up on Linux no matter what, this pr is to get those encoders on Linux.

This PR doesn't touch anything to do with Windows. The encoder implementations are platform specific, and unrelated to each other.

[tytan652]

To be able to use this feature, the user will need to switch to the AMD proprietary driver stack or play between two driver (mesa/amdpro). Which is a bad thing UI/UX wise.

Moreover the feature is not possible on Flatpak.

If the feature is not usable without some workaround, this PR has design issues.

[nowrep]

To be able to use this feature, the user will need to switch to the AMD proprietary driver stack or play between two driver (mesa/amdpro). Which is a bad thing UI/UX wise.

It can be changed so that it switches to proprietary driver automatically without user having to manually set VK_ICD_FILENAMES.

Moreover the feature is not possible on Flatpak.

As long as the proprietary driver is inside the Flatpak runtime, I see no reason why it wouldn't work. Nvidia proprietary driver works with Flatpak too.

[tytan652]

It can be changed so that it switches to proprietary driver automatically without user having to manually set VK_ICD_FILENAMES.

If you mean having a workaround inside OBS code, it is still design issue. OBS should not manage which Vulkan driver has to be used.

As long as the proprietary driver is inside the Flatpak runtime, I see no reason why it wouldn't work. Nvidia proprietary driver works with Flatpak too.

The proprietary driver is not available on Flatpak (Flathub) unlike Nvidia's. Surely because AMD does not allow redistribution of their binaries on "unsupported platform".

Edit: I may have misunderstood but Flatpak use only drivers included in runtimes and never the host.

[CosmicFusion]

That wouldn't work anyway : the icd files and amf.so files , are placed differently on various Distros and third party packages and scripts , for example on ubuntu and fedora you would ; along side the icd being in /opt/amdgpu-pro rather than the system path , you have the AMF.so files in /opt/amdgpu-pro which need to be loaded (a.k.a loaded via LD_LIBRARY_PATH) , this isn't needed in arch ass arch packages have AMF in the system path.

…

________________________________ From: tytan652 ***@***.***> Sent: Saturday, November 19, 2022 9:39:32 AM To: obsproject/obs-studio ***@***.***> Cc: Ward Nakchbandi (Cosmic Fusion) ***@***.***>; Comment ***@***.***> Subject: Re: [obsproject/obs-studio] obs-ffmpeg: Make AMF encoder work on Linux (PR #7206) It can be changed so that it switches to proprietary driver automatically without user having to manually set VK_ICD_FILENAMES. If you mean having a workaround inside OBS code, it is still design issue. OBS should not manage which Vulkan driver has to be used. As long as the proprietary driver is inside the Flatpak runtime, I see no reason why it wouldn't work. Nvidia proprietary driver works with Flatpak too. The proprietary driver is not available on Flatpak (Flathub) unlike Nvidia's. Surely because AMD does not allow redistribution of their binaries on "unsupported platform". — Reply to this email directly, view it on GitHub<#7206 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AT63FLPVO6W2M3323MLFPPTWJB73JANCNFSM572VZKJA>. You are receiving this because you commented.Message ID: ***@***.***>

[alosarjos]

AFAIK this merge request all it does is enable some features if it detects they are available on the system. The fact that this features are a pain in the ass to get installed is not an OBS or this MR problem. If Flatpak can't get this features is an issue between Flatpak and AMD (Regarding redistribution).

I would not consider this MR to have a Design issue. The real problem is outside of OBS

[CosmicFusion]

I know how to repackage AMD drivers : I repackage for the AUR (cfe varients which was AUR updated to 20.40), fedora, and non LTS ubuntu, but for Flatpak we need to use "extra-sources" functionality to not have redist legal issues, and simply couldn't find any resources to learn flatpak

…

________________________________ From: José Miguel Sarasola ***@***.***> Sent: Saturday, November 19, 2022 10:11:20 AM To: obsproject/obs-studio ***@***.***> Cc: Ward Nakchbandi (Cosmic Fusion) ***@***.***>; Comment ***@***.***> Subject: Re: [obsproject/obs-studio] obs-ffmpeg: Make AMF encoder work on Linux (PR #7206) AFAIK this merge request all it does is enable some features if it detects they are available on the system. The fact that this features are a pain in the ass to get installed is not an OBS or this MR problem. If Flatpak can't get this features is an issue between Flatpak and AMD (Regarding redistribution). I would not consider this MR to have a Design issue. The real problem is outside of OBS — Reply to this email directly, view it on GitHub<#7206 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AT63FLMX5WU3HO4CCF6TLGDWJCDSRANCNFSM572VZKJA>. You are receiving this because you commented.Message ID: ***@***.***>

[tytan652]

This AUR package provides a custom desktop entry that apply the workaround, which is also a workaround in itself.

This package is in a way providing biased feedback about this PR, because of this desktop entry workaround.

[Fenrirthviti]

I'm going to go ahead and lock comments for now, as this has derailed from conversation about the actual PR, and is nothing but a mess of troubleshooting and questions about "how do I do X thing".

If we have any questions for the PR author, we will comment here and open comments again. If anyone has any comments on the PR itself, please stop by our #development channel in our Discord, and those conversations can happen there.

We see this PR, and will review it as soon as we have time.