AMD RDNA 4 (RX 9000 series): VCE/VCN hardware encoding not working on Linux

[Kaschtelator]

System

• GPU: AMD Radeon RX 9060 XT (RDNA 4, gfx1200)
• OS: CachyOS (Arch-based), Linux 7.0
• Mesa: 26.1.1
• HandBrake: handbrake-full 1.11.1 (AUR, built with --enable-vce)
• amf-amdgpu-pro-25.10 installed

Problem

H.265 (AMD VCE) and other AMD VCE encoders appear in the encoder list,
but encoding always falls back to CPU. The GPU is never used.

vainfo confirms full VA-API support including hevc_vaapi encoding:
VAProfileHEVCMain: VAEntrypointEncSlice
VAProfileHEVCMain10: VAEntrypointEncSlice
VAProfileAV1Profile0: VAEntrypointEncSlice

ldd /usr/bin/ghb | grep amf returns nothing – HandBrake is not linked
against AMF at runtime.

Root cause

amf-amdgpu-pro-25.10 explicitly supports only RDNA1 & RDNA2.
RDNA 4 is not supported by any available AMF package.
HandBrake does not support VA-API as a fallback for AMD GPUs on Linux.

Request

Please add VA-API support for AMD GPU encoding on Linux (PR #7467 exists
but is not yet merged), or update AMF support to work with RDNA 4 / Mesa.

[github-actions]

Hello,

Welcome to the HandBrake repository.

Warning

Please be cautious of any non-members suggesting that you download a "DLL" or any other files from 3rd party websites.
These are likely malicious!
Members of the HandBrake team can be identified by the "(Member)" label in the top right of any response from them.

Tip

For HandBrake documentation, please see https://handbrake.fr/docs

Thank You,
The HandBrake Bot

[galad87]

What does "encoding always falls back to CPU" means? That's not how it works, please post the activity log.

[Kaschtelator]

Here is the log and a screenshot of the system monitor during encoding.

Activity.log

Was bedeutet „die Kodierung fällt immer auf die CPU zurück“? So funktioniert das nicht, bitte posten Sie das Aktivitätsprotokoll.

[galad87]

The log says it's using the AMD VCN encoder. However, the CPU is running both the decoder and the scale filter, so that's the reason for the high CPU usage.

#7698 will implement the support for the AMD HW decoder and scaler.

[Kaschtelator]

Thanks

Im Protokoll steht, dass der AMD VCN-Encoder verwendet wird. Allerdings läuft auf der CPU sowohl der Decoder als auch der Skalenfilter, das ist also der Grund für die hohe CPU-Auslastung.

#7698 wird die Unterstützung für den AMD HW-Decoder und -Scaler implementieren.