macOS VT HEVC -12900; iOS h264/h265=false

[alexcz-a11y]

Bug Description

• On macOS (controlled side, RustDesk 1.4.2), the internal hwcodec checker fails to initialize HEVC hardware encoder with VideoToolbox:
  • Error setting bitrate property: -12900 (likely kVTPropertyNotSupportedErr)
  • avcodec_open2 failed for hevc_videotoolbox
  • HwCodecConfig ends up with ram_encode: []
• On iOS (controlling side, App Store stable), the client advertises h264=false and h265=false in peer info.
  • Session negotiation picks only VP8/AV1; H.264/H.265 options disappear in the in-session Codec menu.
• System-level VideoToolbox encoding on the same macOS host works fine via ffmpeg for both HEVC and H.264 (yuv420p/I420).
• Attempting to force-enable H.264 VT in RustDesk_hwcodec.toml (ram_encode) does not change the outcome: the checker still tests HEVC first, fails, and leaves ram_encode empty (no H.264 encoder test observed).

How to Reproduce

1. On macOS (controlled side):

   • Install/run RustDesk 1.4.2.
   • Settings → General → enable Hardware codec.

2. On iOS (controlling side, App Store stable):

   • Connect to the macOS host.
   • In-session settings: set color format to I420; set encoder preference to Auto.

3. Observe macOS server logs during session:

   • peer info supported_encoding shows h264=false, h265=false.
   • usable: vp8=true, av1=true, h264=false, h265=false → AV1 selected.

4. Observe macOS hwcodec checker logs on startup:

   • HEVC VT init fails with -12900; avcodec_open2 failed; HwCodecConfig → ram_encode: [].

5. Optional:

   • Edit ~/Library/Preferences/com.carriez.RustDesk/RustDesk_hwcodec.toml to add an H.264 VT entry to ram_encode.
   • Fully quit and restart RustDesk, reconnect from iOS.
   • Result remains the same (no H.264 encoder test, ram_encode stays empty).

Expected Behavior

• macOS side should detect and allow at least one working hardware encoder (e.g., H.264 VT if HEVC fails).
• iOS client should advertise h264/h265=true when supported, so negotiation can select H.264/H.265 instead of falling back to VP8/AV1.

Operating system(s) on local (controlling) side and remote (controlled) side

Local (controlling): iOS (App Store stable; exact version ios26) Remote (controlled): macOS (26)

RustDesk Version(s) on local (controlling) side and remote (controlled) side

Local (iOS): App Store stable (exact version 1.4.2) Remote (macOS): 1.4.2

Screenshots

Additional Context

Key logs (macOS):

• hwcodec checker: ~/Library/Logs/RustDesk/check-hwcodec-config/RustDesk_rCURRENT.log
  [2025-09-30 02:13:17.422925 +08:00] DEBUG ... encode.rs:310] Testing encoder: hevc_videotoolbox
  [2025-09-30 02:13:17.434378 +08:00] ERROR ... ffmpeg.rs:34] Error setting bitrate property: -12900
  [2025-09-30 02:13:17.434696 +08:00] ERROR ... lib.rs:17] [FFMPEG_RAM_ENC] avcodec_open2 failed, ... hevc_videotoolbox
  [2025-09-30 02:13:17.559125 +08:00] DEBUG ... hwcodec.rs:713] HwCodecConfig { signature: 16843009, ram_encode: [], ram_decode: [ ... AV_HWDEVICE_TYPE_VIDEOTOOLBOX ... ] }

• server during iOS→macOS session: ~/Library/Logs/RustDesk/server/RustDesk_rCURRENT.log
  [2025-09-30 02:13:23.985637 +08:00] usable: vp8=true, av1=true, h264=false, h265=false
  [2025-09-30 02:13:23.985646 +08:00] used preference: Auto, encoder: AV1

ffmpeg confirmations (same macOS host, both succeed with I420/yuv420p):
ffmpeg -y -f lavfi -i testsrc2=size=1280x720:rate=30 -t 3 -c:v hevc_videotoolbox -b:v 4M -tag:v hvc1 -pix_fmt yuv420p ./vt_hevc_test.mp4
ffmpeg -y -f lavfi -i testsrc2=size=1280x720:rate=30 -t 3 -c:v h264_videotoolbox -b:v 4M -pix_fmt yuv420p ./vt_h264_test.mp4

Config attempt (macOS): ~/Library/Preferences/com.carriez.RustDesk/RustDesk_hwcodec.toml
{
"signature": 16843009,
"ram_encode": [
{"name":"h264","format":"H264","priority":0,"hwdevice":"AV_HWDEVICE_TYPE_VIDEOTOOLBOX"},
{"name":"hevc","format":"H265","priority":3,"hwdevice":"AV_HWDEVICE_TYPE_VIDEOTOOLBOX"}
],
"ram_decode": [
{"name":"h264","format":"H264","priority":0,"hwdevice":"AV_HWDEVICE_TYPE_VIDEOTOOLBOX"},
{"name":"hevc","format":"H265","priority":0,"hwdevice":"AV_HWDEVICE_TYPE_VIDEOTOOLBOX"},
{"name":"h264","format":"H264","priority":3,"hwdevice":"AV_HWDEVICE_TYPE_NONE"},
{"name":"hevc","format":"H265","priority":3,"hwdevice":"AV_HWDEVICE_TYPE_NONE"}
]
}

Notes / Suggestions:

• On macOS, the checker tries HEVC VT first, fails on a VT property (-12900), and leaves ram_encode empty without testing H.264 VT.
• iOS stable client advertises h264/h265=false, removing H.264/H.265 from negotiation.
• Request:
  • Try H.264 VT when HEVC VT init fails (or provide a toggle to prefer H.264).
  • Consider ignoring/relaxing unsupported VT properties that trigger -12900 on HEVC init.
  • Ensure iOS stable advertises H.264/H.265 when supported so negotiation can select them.

[ody]

I also noticed this issue today, while attempting to connect to my Mac Studio M2 from my ThinkPad AMD P14s running Arch Linux. The laptop kept choosing AV1 as the best codec but I know the M2 does not support AV1 so went looking through logs. Not having hardware encoding makes connecting to a Mac less smooth than other platforms.

[rustdesk]

You can try out this? #12792 (comment) There is a fix.