Segfault in va_openDriver(): ctx->native_dpy is NULL

[vredez]

When using freerdp with VAAPI acceleration, a null pointer occurs in va_openDriver() (see callstack 3-5):

(gdb) bt
#0  0x00007fb5bcc9f615 in raise () from /usr/bin/../lib/libc.so.6
#1  0x00007fb5bd0d023e in ?? () from /usr/bin/../lib/libfreerdp2.so.2
#2  <signal handler called>
#3  XDisplayString (dpy=0x0) at Macros.c:119
#4  0x00007fb5b45c38a7 in __vaDriverInit_1_4 () from /usr/lib/dri/vdpau_drv_video.so
#5  0x00007fb5b82c4c39 in va_openDriver (dpy=dpy@entry=0x7fb5b0337660, driver_name=<optimized out>) at ../libva/va/va.c:517
#6  0x00007fb5b82cac4f in vaInitialize (dpy=dpy@entry=0x7fb5b0337660, major_version=major_version@entry=0x7fb5b6dda4d0,
    minor_version=minor_version@entry=0x7fb5b6dda4d4) at ../libva/va/va.c:737
#7  0x00007fb5bae3e78e in vaapi_device_connect (ctx=0x7fb5b0337240, display=0x7fb5b0337660) at libavutil/hwcontext_vaapi.c:1482
#8  0x00007fb5bae36c4f in av_hwdevice_ctx_create (pdevice_ref=0x7fb5b0329c88, type=<optimized out>, device=0x7fb5bd1e6d97 "/dev/dri/renderD128", opts=0x0, flags=0)
    at libavutil/hwcontext.c:629
#9  0x00007fb5bd1a5fd6 in ?? () from /usr/bin/../lib/libfreerdp2.so.2
#10 0x00007fb5bd1a0906 in h264_context_new () from /usr/bin/../lib/libfreerdp2.so.2
#11 0x00007fb5bd133068 in freerdp_client_codecs_prepare () from /usr/bin/../lib/libfreerdp2.so.2
#12 0x00007fb5bd13a63a in ?? () from /usr/bin/../lib/libfreerdp2.so.2
#13 0x00007fb5bd12daa4 in freerdp_connect () from /usr/bin/../lib/libfreerdp2.so.2
#14 0x000055c50fc3781d in ?? ()
#15 0x00007fb5bcea6732 in ?? () from /usr/bin/../lib/libwinpr2.so.2
#16 0x00007fb5bcc193e9 in start_thread () from /usr/bin/../lib/../lib/libpthread.so.0
#17 0x00007fb5bcd62293 in clone () from /usr/bin/../lib/libc.so.6

The X11 display handle (ctx->native_dpy) is not initialized, which leads to a segfault in XDisplayString().

As a dirty quick fix I applied following patch which solves the problem:

diff --unified --recursive --text libva/va/va.c libva_patched/va/va.c
--- libva/va/va.c	2020-12-03 18:06:39.480335569 +0100
+++ libva_patched/va/va.c	2020-12-03 17:38:59.001928000 +0100
@@ -422,6 +422,8 @@
 static VAStatus va_openDriver(VADisplay dpy, char *driver_name)
 {
     VADriverContextP ctx = CTX(dpy);
+    ctx->native_dpy = dpy;
+
     VAStatus vaStatus = VA_STATUS_ERROR_UNKNOWN;
     char *search_path = NULL;
     char *saveptr;

I'm sure there is a better place within the code to ensure that the native_dpy is set.

SW

libva: 2.9.1 (I also tested with 2.10.0, same issue here)
libx11: 1.7.0
libva-vdpau-driver: 0.7.4
libvdpau: 1.4
ffmpeg: 4.3.1
freerdp: 2.2.0
nvidia: 455.45.01
xorg-xserver: 1.20.10
linux: 5.9.11-arch2-1 (Arch Linux)

HW

gpu: NVIDIA Corporation TU106 [GeForce RTX 2070] (rev a1)

[XinfengZhang]

have you call vaGetDisplay in the application? suppose vaGetDisplay will send dpy and it will be set to native_dpy

[vredez]

I've investigated this and found out that application actually calls vaGetDisplayDRM(), which apparently is not supported by the underlying VAAPI driver.
Thanks for your help!

[XinfengZhang]

sorry, was the problem resolved? I just checked code of vaGetDisplayDRM(), the naitive_dpy is not set. @xhaihao , do you know the history of this implementation ? @vredez feel free to open the issue

[vredez]

Yes, the problem is resolved. It's no issue of libva, the problem lies within the libva-vdpau-driver, which always assumes the connection type "X11". If the "DRM" connection type is used, there is no native window and thus dpy == NULL. See this FreeRDP issue for more details.