Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues trying out open-h264 #159

Closed
johnae opened this issue Oct 1, 2022 · 21 comments
Closed

Issues trying out open-h264 #159

johnae opened this issue Oct 1, 2022 · 21 comments

Comments

@johnae
Copy link

johnae commented Oct 1, 2022

Thanks for an awesome project!

I've been interested in the h264 encoding but when trying it out the server crashes with this error:

❯ wayvnc -g -v 127.0.0.1 5900 -o DP-2
Info: New client connection: 0xc54ec0 (ref 1)
Info: Choosing raw encoding for client 0xc54ec0
Info: Client 0xc54ec0 (1) hung up
Info: Closing client connection 0xc54ec0: ref 0
Info: New client connection: 0xc54ec0 (ref 1)
Info: Choosing open-h264 encoding for client 0xc54ec0
Info: Opened DRM device /dev/dri/renderD128: driver amdgpu version 3.47.0.
Info: w:1 h:1 pixfmt:drm_prime tb:1/1 fr:0/1 sar:0/1
Info: libva: VA-API version 1.15.0
Info: libva: Trying to open /run/opengl-driver/lib/dri/radeonsi_drv_video.so
Info: libva: Found init function __vaDriverInit_1_15
Info: libva: va_openDriver() returns 0
Info: Initialised VAAPI connection: version 1.15
Info: VAAPI driver: Mesa Gallium driver 22.1.6 for AMD Radeon RX 5600 XT (navi10, LLVM 14.0.6, DRM 3.47, 5.19.8).
Info: Driver not found in known nonstandard list, using standard behaviour.
Info: Input surface format is nv12.
Info: Using VAAPI profile VAProfileH264ConstrainedBaseline (13).
Info: Using VAAPI entrypoint VAEntrypointEncSlice (6).
Info: Using VAAPI render target format YUV420 (0x1).
Info: RC mode: CQP.
Info: RC quality: 20.
Info: RC framerate: 65535/1 (65535.00 fps).
Info: Using intra and P-frames (supported references: 1 / 0).
Warning: libav: 0: Driver does not support some wanted packed headers (wanted 0xd, found 0).
Info: Using level 5.1.
EE ../src/gallium/drivers/radeon/radeon_vcn_enc_1_2.c:1161 radeon_enc_encode_params UVD - DCC surfaces not supported.

Is Radeon 5600 XT not supported or do you think something else may be wrong with my configuration?
@any1
Copy link
Owner

any1 commented Oct 1, 2022

Maybe direct mapping of DRM buffers into VAAPI surfaces isn't supported on your platform. Have you tried running with --log-level=trace instead of -v?

A backtrace with debugging symbols would also be good if it's crashing.

@johnae
Copy link
Author

johnae commented Oct 1, 2022 •
edited
Loading

This is with log-level trace:

❯ wayvnc -g -v 127.0.0.1 5900 -o DP-2 --log-level trace
Info: New client connection: 0x196c030 (ref 1)
Info: Choosing open-h264 encoding for client 0x196c030
Info: Opened DRM device /dev/dri/renderD128: driver amdgpu version 3.47.0.
DEBUG: libav: 0: detected 32 logical cores
DEBUG: libav: 0: Setting 'width' to value '1'
DEBUG: libav: 0: Setting 'height' to value '1'
DEBUG: libav: 0: Setting 'pix_fmt' to value 'drm_prime'
DEBUG: libav: 0: Setting 'time_base' to value '1/1'
Info: w:1 h:1 pixfmt:drm_prime tb:1/1 fr:0/1 sar:0/1
DEBUG: libav: 0: Setting 'mode' to value 'direct'
DEBUG: libav: 0: Setting 'derive_device' to value 'vaapi'
DEBUG: libav: 0: Setting 'format' to value 'nv12'
DEBUG: libav: 0: Setting 'mode' to value 'fast'
DEBUG: libav: 0: query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
DEBUG: libav: 0: Configure hwmap drm_prime -> vaapi_vld.
Info: libva: VA-API version 1.15.0
Info: libva: Trying to open /run/opengl-driver/lib/dri/radeonsi_drv_video.so
Info: libva: Found init function __vaDriverInit_1_15
Info: libva: va_openDriver() returns 0
Info: Initialised VAAPI connection: version 1.15
DEBUG: libav: 0: Format 0x3231564e -> nv12.
DEBUG: libav: 0: Format 0x30313050 -> p010le.
DEBUG: libav: 0: Format 0x36313050 -> unknown.
DEBUG: libav: 0: Format 0x30323449 -> yuv420p.
DEBUG: libav: 0: Format 0x32315659 -> yuv420p.
DEBUG: libav: 0: Format 0x56595559 -> unknown.
DEBUG: libav: 0: Format 0x32595559 -> yuyv422.
DEBUG: libav: 0: Format 0x59565955 -> uyvy422.
DEBUG: libav: 0: Format 0x41524742 -> bgra.
DEBUG: libav: 0: Format 0x41424752 -> rgba.
DEBUG: libav: 0: Format 0x58524742 -> bgr0.
DEBUG: libav: 0: Format 0x58424752 -> rgb0.
Info: VAAPI driver: Mesa Gallium driver 22.1.6 for AMD Radeon RX 5600 XT (navi10, LLVM 14.0.6, DRM 3.47, 5.19.8).
Info: Driver not found in known nonstandard list, using standard behaviour.
DEBUG: libav: 0: Created surface 0x2.
DEBUG: libav: 0: Direct mapping disabled: deriving image does not work: 1 (operation failed).
DEBUG: libav: 0: Created surface 0x3.
DEBUG: libav: 0: Created surface 0x4.
DEBUG: libav: 0: Created surface 0x5.
DEBUG: libav: 0: Created surface 0x6.
DEBUG: libav: 0: Created surface 0x7.
DEBUG: libav: 0: Created surface 0x8.
DEBUG: libav: 0: Created surface 0x9.
DEBUG: libav: 0: Created surface 0xa.
DEBUG: libav: 0: Created surface 0xb.
Info: Input surface format is nv12.
Info: Using VAAPI profile VAProfileH264ConstrainedBaseline (13).
Info: Using VAAPI entrypoint VAEntrypointEncSlice (6).
Info: Using VAAPI render target format YUV420 (0x1).
DEBUG: libav: 0: Driver supports RC modes CQP, CBR, VBR.
DEBUG: libav: 0: Driver does not support ICQ RC mode.
Info: RC mode: CQP.
Info: RC quality: 20.
Info: RC framerate: 65535/1 (65535.00 fps).
Info: Using intra and P-frames (supported references: 1 / 0).
Warning: libav: 0: Driver does not support some wanted packed headers (wanted 0xd, found 0).
DEBUG: libav: 0: Using nv12 as format of reconstructed frames.
DEBUG: libav: 0: Created surface 0xe.
DEBUG: libav: 0: Direct mapping disabled: deriving image does not work: 1 (operation failed).
DEBUG: libav: 0: Using fixed QP = 20 / 20 / 24 for IDR- / P- / B-frames.
Info: Using level 5.1.
DEBUG: libav: 0: Filter input: drm_prime, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Map DRM object 23 to VAAPI as 58524742.
DEBUG: libav: 0: Create surface 0x10.
DEBUG: libav: 0: Mapped DRM object 23 to surface 0x10.
DEBUG: libav: 0: Filter output: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Filter input: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Mapped colour properties unknown gbr/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Mapped colour properties unknown unknown/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Pipeline parameter buffer is 0x11.
DEBUG: libav: 0: Destroy surface 0x10.
DEBUG: libav: 0: Filter output: vaapi_vld, 3840x2160 (-9223372036854775808), mode: FAST.
DEBUG: libav: 0: Input frame: 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Pick forced IDR-picture to encode next.
DEBUG: libav: 0: Issuing encode for pic 0/0 as type IDR.
DEBUG: libav: 0: No reference pictures.
DEBUG: libav: 0: Input surface is 0xb.
DEBUG: libav: 0: Recon surface is 0xe.
DEBUG: libav: 0: Allocated output buffer 0x10
DEBUG: libav: 0: Output buffer is 0x10.
DEBUG: libav: 0: Param buffer (22) is 0x11.
DEBUG: libav: 0: Param buffer (27) is 0x12.
DEBUG: libav: 0: Param buffer (23) is 0x13.
DEBUG: libav: 0: Slice 0: 0-134 (135 rows), 0-32399 (32400 blocks).
DEBUG: libav: 0: Param buffer (24) is 0x14.
EE ../src/gallium/drivers/radeon/radeon_vcn_enc_1_2.c:1161 radeon_enc_encode_params UVD - DCC surfaces not supported.
DEBUG: libav: 0: Sync to pic 0/0 (input surface 0xb).
DEBUG: libav: 0: Output buffer: 1008472 bytes (status 00000000).
DEBUG: libav: 0: Output read for pic 0/0.
DEBUG: libav: 0: Output packet: pts -9223372036854775808 dts -9223372036854775808.
DEBUG: libav: 0: Filter input: drm_prime, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Map DRM object 23 to VAAPI as 58524742.
DEBUG: libav: 0: Create surface 0x11.
DEBUG: libav: 0: Mapped DRM object 23 to surface 0x11.
DEBUG: libav: 0: Filter output: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Filter input: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Mapped colour properties unknown gbr/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Mapped colour properties unknown unknown/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Pipeline parameter buffer is 0x12.

@johnae
Copy link
Author

johnae commented Oct 1, 2022 •
edited
Loading

W.r.t my platform I'm on:

Linux eris 5.19.8 #1-NixOS SMP PREEMPT_DYNAMIC Thu Sep 8 09:24:08 UTC 2022 x86_64 GNU/Linux - NixOS is my distribution.

❯ vainfo
libva info: VA-API version 1.15.0
libva info: Trying to open /run/opengl-driver/lib/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_15
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.15 (libva 2.15.0)
vainfo: Driver version: Mesa Gallium driver 22.1.6 for AMD Radeon RX 5600 XT (navi10, LLVM 14.0.6, DRM 3.47, 5.19.8)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc

@any1
Copy link
Owner

any1 commented Oct 2, 2022

As far as I understand "DCC" is some sort of texture compression and the encoder just doesn't know how to handle compressed buffers.

I suspect that buffers that must be scanoutable won't be compressed in this way, so you could try applying the following:

diff --git a/src/buffer.c b/src/buffer.c
index 3811cde..5a405d4 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -140,7 +140,7 @@ static struct wv_buffer* wv_buffer_create_dmabuf(int width, int height,
        self->format = fourcc;
 
        self->bo = gbm_bo_create(gbm_device, width, height, fourcc,
-                       GBM_BO_USE_RENDERING);
+                       GBM_BO_USE_RENDERING | GBM_BO_USE_SCANOUT);
        if (!self->bo)
                goto bo_failure;

If that fails, try this instead:

diff --git a/src/buffer.c b/src/buffer.c
index 3811cde..19a52cd 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -140,7 +140,7 @@ static struct wv_buffer* wv_buffer_create_dmabuf(int width, int height,
 	self->format = fourcc;
 
 	self->bo = gbm_bo_create(gbm_device, width, height, fourcc,
-			GBM_BO_USE_RENDERING);
+			GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
 	if (!self->bo)
 		goto bo_failure;

@johnae
Copy link
Author

johnae commented Oct 2, 2022

With either of those two patches I actually get an image displayed in wlvncc but immediately after making the connection, wayvnc exits:

❯ ./result/bin/wayvnc -g -o DP-2 127.0.0.1 5900 --log-level trace
Info: New client connection: 0x1101510 (ref 1)
Info: Starting screen capture
DEBUG: ../src/main.c: 826: Client connected, new client count: 1
Info: Choosing open-h264 encoding for client 0x1101510
Info: Opened DRM device /dev/dri/renderD128: driver amdgpu version 3.47.0.
DEBUG: libav: 0: detected 32 logical cores
DEBUG: libav: 0: Setting 'width' to value '1'
DEBUG: libav: 0: Setting 'height' to value '1'
DEBUG: libav: 0: Setting 'pix_fmt' to value 'drm_prime'
DEBUG: libav: 0: Setting 'time_base' to value '1/1'
Info: w:1 h:1 pixfmt:drm_prime tb:1/1 fr:0/1 sar:0/1
DEBUG: libav: 0: Setting 'mode' to value 'direct'
DEBUG: libav: 0: Setting 'derive_device' to value 'vaapi'
DEBUG: libav: 0: Setting 'format' to value 'nv12'
DEBUG: libav: 0: Setting 'mode' to value 'fast'
DEBUG: libav: 0: query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
DEBUG: libav: 0: Configure hwmap drm_prime -> vaapi_vld.
Info: libva: VA-API version 1.15.0
Info: libva: Trying to open /run/opengl-driver/lib/dri/radeonsi_drv_video.so
Info: libva: Found init function __vaDriverInit_1_15
Info: libva: va_openDriver() returns 0
Info: Initialised VAAPI connection: version 1.15
DEBUG: libav: 0: Format 0x3231564e -> nv12.
DEBUG: libav: 0: Format 0x30313050 -> p010le.
DEBUG: libav: 0: Format 0x36313050 -> unknown.
DEBUG: libav: 0: Format 0x30323449 -> yuv420p.
DEBUG: libav: 0: Format 0x32315659 -> yuv420p.
DEBUG: libav: 0: Format 0x56595559 -> unknown.
DEBUG: libav: 0: Format 0x32595559 -> yuyv422.
DEBUG: libav: 0: Format 0x59565955 -> uyvy422.
DEBUG: libav: 0: Format 0x41524742 -> bgra.
DEBUG: libav: 0: Format 0x41424752 -> rgba.
DEBUG: libav: 0: Format 0x58524742 -> bgr0.
DEBUG: libav: 0: Format 0x58424752 -> rgb0.
Info: VAAPI driver: Mesa Gallium driver 22.1.6 for AMD Radeon RX 5600 XT (navi10, LLVM 14.0.6, DRM 3.47, 5.19.8).
Info: Driver not found in known nonstandard list, using standard behaviour.
DEBUG: libav: 0: Created surface 0x2.
DEBUG: libav: 0: Direct mapping disabled: deriving image does not work: 1 (operation failed).
DEBUG: libav: 0: Created surface 0x3.
DEBUG: libav: 0: Created surface 0x4.
DEBUG: libav: 0: Created surface 0x5.
DEBUG: libav: 0: Created surface 0x6.
DEBUG: libav: 0: Created surface 0x7.
DEBUG: libav: 0: Created surface 0x8.
DEBUG: libav: 0: Created surface 0x9.
DEBUG: libav: 0: Created surface 0xa.
DEBUG: libav: 0: Created surface 0xb.
Info: Input surface format is nv12.
Info: Using VAAPI profile VAProfileH264ConstrainedBaseline (13).
Info: Using VAAPI entrypoint VAEntrypointEncSlice (6).
Info: Using VAAPI render target format YUV420 (0x1).
DEBUG: libav: 0: Driver supports RC modes CQP, CBR, VBR.
DEBUG: libav: 0: Driver does not support ICQ RC mode.
Info: RC mode: CQP.
Info: RC quality: 23.
Info: RC framerate: 65535/1 (65535.00 fps).
Info: Using intra and P-frames (supported references: 1 / 0).
Warning: libav: 0: Driver does not support some wanted packed headers (wanted 0xd, found 0).
DEBUG: libav: 0: Using nv12 as format of reconstructed frames.
DEBUG: libav: 0: Created surface 0xe.
DEBUG: libav: 0: Direct mapping disabled: deriving image does not work: 1 (operation failed).
DEBUG: libav: 0: Using fixed QP = 23 / 23 / 28 for IDR- / P- / B-frames.
Info: Using level 5.1.
DEBUG: libav: 0: Filter input: drm_prime, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Map DRM object 20 to VAAPI as 58524742.
DEBUG: libav: 0: Create surface 0x10.
DEBUG: libav: 0: Mapped DRM object 20 to surface 0x10.
DEBUG: libav: 0: Filter output: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Filter input: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Mapped colour properties unknown gbr/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Mapped colour properties unknown unknown/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Pipeline parameter buffer is 0x11.
DEBUG: libav: 0: Destroy surface 0x10.
DEBUG: libav: 0: Filter output: vaapi_vld, 3840x2160 (-9223372036854775808), mode: FAST.
DEBUG: libav: 0: Input frame: 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Pick forced IDR-picture to encode next.
DEBUG: libav: 0: Issuing encode for pic 0/0 as type IDR.
DEBUG: libav: 0: No reference pictures.
DEBUG: libav: 0: Input surface is 0xb.
DEBUG: libav: 0: Recon surface is 0xe.
DEBUG: libav: 0: Allocated output buffer 0x10
DEBUG: libav: 0: Output buffer is 0x10.
DEBUG: libav: 0: Param buffer (22) is 0x11.
DEBUG: libav: 0: Param buffer (27) is 0x12.
DEBUG: libav: 0: Param buffer (23) is 0x13.
DEBUG: libav: 0: Slice 0: 0-134 (135 rows), 0-32399 (32400 blocks).
DEBUG: libav: 0: Param buffer (24) is 0x14.
DEBUG: libav: 0: Sync to pic 0/0 (input surface 0xb).
DEBUG: libav: 0: Output buffer: 475840 bytes (status 00000000).
DEBUG: libav: 0: Output read for pic 0/0.
DEBUG: libav: 0: Output packet: pts -9223372036854775808 dts -9223372036854775808.
DEBUG: libav: 0: Filter input: drm_prime, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Map DRM object 20 to VAAPI as 58524742.
DEBUG: libav: 0: Create surface 0x11.
DEBUG: libav: 0: Mapped DRM object 20 to surface 0x11.
DEBUG: libav: 0: Filter output: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Filter input: vaapi_vld, 3840x2160 (-9223372036854775808).
DEBUG: libav: 0: Mapped colour properties unknown gbr/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Mapped colour properties unknown unknown/unknown/unknown unspecified to VA standard 0 chroma siting 0 range 0.
DEBUG: libav: 0: Pipeline parameter buffer is 0x12.

Seems the warning I get has been mentioned elsewhere, no idea if it's related though:

ammen99/wf-recorder#44
https://gitlab.freedesktop.org/mesa/mesa/-/issues/2351

@any1
Copy link
Owner

any1 commented Oct 3, 2022

I don't think that warning is the problem here. Can you try with TigerVNC master? This might just as well be a problem with wlvncc.

@any1
Copy link
Owner

any1 commented Oct 3, 2022

Hmm, actually, wayvnc shouldn't just exit. That's weird. The only way for it to just exit without logging something out is if it receives SIGINT or if some library that it's using calls exit() directly.

To rule out SIGINT, can you try this?

index d5b29cf..0bd8e0f 100644
--- a/src/main.c
+++ b/src/main.c
@@ -424,6 +424,7 @@ void wayvnc_exit(struct wayvnc* self)
 void on_signal(void* obj)
 {
        struct wayvnc* self = aml_get_userdata(obj);
+       nvnc_log(NVNC_LOG_INFO, "Received signal. Exiting...");
        wayvnc_exit(self);
 }

@johnae
Copy link
Author

johnae commented Oct 3, 2022

Actually, it's a Segmentation fault (core dumped). No idea why I missed that before.

@any1
Copy link
Owner

any1 commented Oct 3, 2022

Well, in that case a backtrace with debug symbols would be useful.

@johnae
Copy link
Author

johnae commented Oct 3, 2022

I'm afraid I don't have debug symbols for all dependencies here (on NixOS that can be somewhat of an exercise), anyway - this is what I have so far:

(gdb) bt full
#0  0x00007f679666485c in vl_video_buffer_sampler_view_components () from /run/opengl-driver/lib/dri/radeonsi_drv_video.so
No symbol table info available.
#1  0x00007f6796658c20 in vl_compositor_set_buffer_layer () from /run/opengl-driver/lib/dri/radeonsi_drv_video.so
No symbol table info available.
#2  0x00007f67966073f2 in vlVaHandleVAProcPipelineParameterBufferType () from /run/opengl-driver/lib/dri/radeonsi_drv_video.so
No symbol table info available.
#3  0x00007f6796602129 in vlVaRenderPicture () from /run/opengl-driver/lib/dri/radeonsi_drv_video.so
No symbol table info available.
#4  0x00007f686ff7f882 in vaRenderPicture () from /nix/store/ia2kkn965r5pqsfyhf4lmlik7ck5vw3i-libva-2.15.0/lib/libva.so.2
No symbol table info available.
#5  0x00007f687267dcfc in ff_vaapi_vpp_render_picture (avctx=avctx@entry=0x23a1540, params=params@entry=0x7f67d19be4c0, output_frame=<optimized out>) at libavfilter/vaapi_vpp.c:625
        ctx = 0x23a1680
        output_surface = <optimized out>
        params_id = 18
        vas = 0
        err = <optimized out>
        fail = <optimized out>
#6  0x00007f6872773aaf in scale_vaapi_filter_frame (inlink=inlink@entry=0x23a1e40, input_frame=<optimized out>) at libavfilter/vf_scale_vaapi.c:148
        avctx = 0x23a1540
        outlink = 0x23a2280
        vpp_ctx = 0x23a1680
        ctx = 0x23a1680
        output_frame = 0x7f66e0002880
        params = {surface = 17, surface_region = 0x23a16b8, surface_color_standard = VAProcColorStandardNone, output_region = 0x0, output_background_color = 4278190080, output_color_standard = VAProcColorStandardNone, pipeline_flags = 0, filter_flags = 256, filters = 0x0, num_filters = 0, forward_references = 0x0, num_forward_references = 0,
          backward_references = 0x0, num_backward_references = 0, rotation_state = 0, blend_state = 0x0, mirror_state = 0, additional_outputs = 0x0, num_additional_outputs = 0, input_surface_flag = 0, output_surface_flag = 0, input_color_properties = {chroma_sample_location = 0 '\000', color_range = 0 '\000', colour_primaries = 2 '\002',
            transfer_characteristics = 2 '\002', matrix_coefficients = 0 '\000', reserved = "\000\000"}, output_color_properties = {chroma_sample_location = 0 '\000', color_range = 0 '\000', colour_primaries = 2 '\002', transfer_characteristics = 2 '\002', matrix_coefficients = 2 '\002', reserved = "\000\000"},
          processing_mode = VAProcDefaultMode, output_hdr_metadata = 0x0, va_reserved = {0 <repeats 16 times>}}
        err = 0
#7  0x00007f687264f999 in ff_filter_frame_framed (frame=<optimized out>, link=0x23a1e40) at libavfilter/avfilter.c:1087
        filter_frame = 0x7f6872773960 <scale_vaapi_filter_frame>
        dstctx = 0x23a1540
        dst = <optimized out>
        ret = <optimized out>
        filter_frame = <optimized out>
        dstctx = <optimized out>
        dst = <optimized out>
        ret = <optimized out>
        fail = <optimized out>
#8  ff_filter_frame_to_filter (link=0x23a1e40) at libavfilter/avfilter.c:1235
        frame = 0x7f66e0001840
        dst = 0x23a1540
        ret = <optimized out>
        frame = <optimized out>
        dst = <optimized out>
        ret = <optimized out>
#9  ff_filter_activate_default (filter=<optimized out>) at libavfilter/avfilter.c:1284
        i = <optimized out>
        i = <optimized out>
#10 ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1443
        ret = <optimized out>
#11 0x00007f68726525a7 in ff_filter_graph_run_once (graph=<optimized out>) at libavfilter/avfiltergraph.c:1403
        filter = <optimized out>
        i = <optimized out>
#12 0x00007f687265356d in get_frame_internal (ctx=0x23a03c0, frame=0x7f66e00015c0, flags=flags@entry=0, samples=<optimized out>) at libavfilter/buffersink.c:131
        buf = 0x23a04c0
        inlink = 0x23a2280
        status = 0
        ret = <optimized out>
        cur_frame = 0x0
        pts = -9223372036854775808
#13 0x00007f6872653732 in av_buffersink_get_frame_flags (ctx=<optimized out>, frame=<optimized out>, flags=flags@entry=0) at libavfilter/buffersink.c:142
No locals.
#14 0x00007f6872653747 in av_buffersink_get_frame (ctx=<optimized out>, frame=<optimized out>) at libavfilter/buffersink.c:90
No locals.
#15 0x00007f687422d115 in h264_encoder__encode (frame_in=<optimized out>, self=0x23886f0) at ../src/h264-encoder.c:368
        packet = 0x7f66e0000dc0
        rc = <optimized out>
        filtered_frame = 0x7f66e00015c0
        rc = <optimized out>
        filtered_frame = <optimized out>
        get_frame_failure = <optimized out>
        send_frame_failure = <optimized out>
        packet = <optimized out>
        __PRETTY_FUNCTION__ = "h264_encoder__encode"
#16 h264_encoder__do_work (handle=<optimized out>) at ../src/h264-encoder.c:419
        self = 0x23886f0
        frame = 0x7f66e0000dc0
        __PRETTY_FUNCTION__ = "h264_encoder__do_work"
        rc = <optimized out>
#17 0x00007f6874279a58 in worker_fn () from /nix/store/7a4hwar6jq5hbdzfdcrk225jniw1n7z6-aml-3afc3aacce6d0342bdff71c38cc477d6e23b9be3/lib/libaml.so.0
No symbol table info available.

@any1
Copy link
Owner

any1 commented Oct 3, 2022

All I can tell from this is that it's crashing inside the VA driver in mesa. There's a chance we can figure this out with debug symbols from mesa. I recommend compiling things with the -Og flag so that things don't get <optimized out>.

I can't say if it's a driver bug or if the supplied buffers are misconfigured.

@johnae
Copy link
Author

johnae commented Oct 4, 2022

I believe I've got better output now but as noted, I'm not entirely familiar with how to get debugging symbols and disable optimization on NixOS:

(gdb) bt full
#0  0x00007fea3ab93d0e in vl_video_buffer_sampler_view_components (buffer=0x7fe994002170) at ../src/gallium/auxiliary/vl/vl_video_buffer.c:272
        res = <optimized out>
        desc = <optimized out>
        nr_components = <optimized out>
        buf = 0x7fe994002170
        sv_templ = {{__cl_space = "\001\000\000\000\000\000\000\000\030\000\000\000\060\000\000\000\200\322\177_\352\177\000\000\300\321\177_\352\177\000\000\a\000\000\000\000\000\000\000\220\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\005\372\255\f\353\177\000", reference = {count = 1}}, format = 31237, target = 27,
          swizzle_r = 2, swizzle_g = 1, swizzle_b = 3, swizzle_a = 0, texture = 0x7feb0cadfa05, context = 0xfffffffffffffe38, u = {tex = {first_layer = 2, last_layer = 0, first_level = 0, last_level = 0}, buf = {offset = 2, size = 0}}}
        pipe = 0x16c4c20
        sampler_format = {PIPE_FORMAT_B8G8R8X8_UNORM, PIPE_FORMAT_NONE, PIPE_FORMAT_NONE}
        plane_order = 0x0
        i = 0
        j = <optimized out>
        component = 0
#1  0x00007fea3ab863b8 in vl_compositor_set_buffer_layer (s=s@entry=0x166d888, c=c@entry=0x166d748, layer=layer@entry=0, buffer=buffer@entry=0x7fe994002170, src_rect=src_rect@entry=0x7fea5f7fd2d0, dst_rect=dst_rect@entry=0x0, deinterlace=VL_COMPOSITOR_NONE) at ../src/gallium/auxiliary/vl/vl_compositor.c:562
        sampler_views = <optimized out>
        i = <optimized out>
#2  0x00007fea3ab3c6d3 in vlVaPostProcCompositor (drv=drv@entry=0x166d730, context=context@entry=0x18bc2e0, src_region=src_region@entry=0x1656d78, dst_region=<optimized out>, src=src@entry=0x7fe994002170, dst=<optimized out>, deinterlace=VL_COMPOSITOR_NONE) at ../src/gallium/frontends/va/postproc.c:81
        surfaces = 0x7fe9900034e8
        src_rect = {x0 = 0, x1 = 3840, y0 = 0, y1 = 2160}
        dst_rect = {x0 = 0, x1 = 3840, y0 = 0, y1 = 2160}
#3  0x00007fea3ab3d15f in vlVaHandleVAProcPipelineParameterBufferType (drv=drv@entry=0x166d730, context=context@entry=0x18bc2e0, buf=<optimized out>) at ../src/gallium/frontends/va/postproc.c:408
        deinterlace = VL_COMPOSITOR_NONE
        def_src_region = {x = 10426, y = 3242, width = 32747, height = 0}
        def_dst_region = {x = 0, y = 0, width = 3840, height = 2160}
        src_region = 0x1656d78
        dst_region = <optimized out>
        param = 0x7fe994002d20
        src = 0x7fe994002170
        dst = <optimized out>
        src_surface = 0x7fe994001b40
        dst_surface = 0x18c44c0
        i = <optimized out>
        pscreen = <optimized out>
#4  0x00007fea3ab381e9 in vlVaRenderPicture (ctx=<optimized out>, context_id=<optimized out>, buffers=0x7fea5f7fd42c, num_buffers=1) at ../src/gallium/frontends/va/picture.c:618
        buf = <optimized out>
        drv = 0x166d730
        context = 0x18bc2e0
        vaStatus = 0
        i = 0
        slice_param_idx = 0
#5  0x00007feb0a768882 in vaRenderPicture () from /nix/store/ia2kkn965r5pqsfyhf4lmlik7ck5vw3i-libva-2.15.0/lib/libva.so.2
No symbol table info available.
#6  0x00007feb0ce1c94f in ff_vaapi_vpp_render_picture (avctx=avctx@entry=0x1656c40, params=params@entry=0x7fea5f7fd470, output_frame=<optimized out>) at libavfilter/vaapi_vpp.c:625
        ctx = 0x1656d40
        output_surface = <optimized out>
        params_id = 18
        vas = 0
        err = <optimized out>
        fail = <optimized out>
#7  0x00007feb0ceede84 in scale_vaapi_filter_frame (inlink=inlink@entry=0x1657500, input_frame=<optimized out>) at libavfilter/vf_scale_vaapi.c:148
        avctx = 0x1656c40
        outlink = 0x1657940
        vpp_ctx = 0x1656d40
        ctx = 0x1656d40
        output_frame = 0x7fe994002880
        params = {surface = 17, surface_region = 0x1656d78, surface_color_standard = VAProcColorStandardNone, output_region = 0x0, output_background_color = 4278190080, output_color_standard = VAProcColorStandardNone, pipeline_flags = 0, filter_flags = 256, filters = 0x0, num_filters = 0, forward_references = 0x0, num_forward_references = 0,
          backward_references = 0x0, num_backward_references = 0, rotation_state = 0, blend_state = 0x0, mirror_state = 0, additional_outputs = 0x0, num_additional_outputs = 0, input_surface_flag = 0, output_surface_flag = 0, input_color_properties = {chroma_sample_location = 0 '\000', color_range = 0 '\000', colour_primaries = 2 '\002',
            transfer_characteristics = 2 '\002', matrix_coefficients = 0 '\000', reserved = "\000\000"}, output_color_properties = {chroma_sample_location = 0 '\000', color_range = 0 '\000', colour_primaries = 2 '\002', transfer_characteristics = 2 '\002', matrix_coefficients = 2 '\002', reserved = "\000\000"},
          processing_mode = VAProcDefaultMode, output_hdr_metadata = 0x0, va_reserved = {0 <repeats 16 times>}}
        err = 0
#8  0x00007feb0cdf2f7d in ff_filter_frame_framed (link=link@entry=0x1657500, frame=<optimized out>) at libavfilter/avfilter.c:1087
        filter_frame = 0x7feb0ceedd11 <scale_vaapi_filter_frame>
        dstctx = 0x1656c40
        dst = <optimized out>
        ret = <optimized out>
#9  0x00007feb0cdf315d in ff_filter_frame_to_filter (link=link@entry=0x1657500) at libavfilter/avfilter.c:1235
        frame = 0x7fe994001840
        dst = 0x1656c40
        ret = 1
#10 0x00007feb0cdf31e5 in ff_filter_activate_default (filter=0x1656c40) at libavfilter/avfilter.c:1284
        i = 0
#11 0x00007feb0cdf3285 in ff_filter_activate (filter=<optimized out>) at libavfilter/avfilter.c:1443
        ret = <optimized out>
#12 0x00007feb0cdf5b04 in ff_filter_graph_run_once (graph=<optimized out>) at libavfilter/avfiltergraph.c:1403
        filter = <optimized out>
        i = <optimized out>
#13 0x00007feb0cdf63c1 in get_frame_internal (ctx=0x1655a80, frame=0x7fe9940015c0, flags=flags@entry=0, samples=0) at libavfilter/buffersink.c:131
        buf = 0x1655bc0
        inlink = 0x1657940
        status = 0
        ret = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        cur_frame = 0x0
        pts = -9223372036854775808
#14 0x00007feb0cdf6867 in av_buffersink_get_frame_flags (ctx=<optimized out>, frame=<optimized out>, flags=flags@entry=0) at libavfilter/buffersink.c:142
No locals.
#15 0x00007feb0cdf687a in av_buffersink_get_frame (ctx=<optimized out>, frame=<optimized out>) at libavfilter/buffersink.c:90
No locals.
#16 0x00007feb0e7ab0f6 in h264_encoder__encode (self=self@entry=0x163c9e0, frame_in=<optimized out>) at ../src/h264-encoder.c:368
        rc = 0
        filtered_frame = 0x7fe9940015c0
        packet = 0x0
        __PRETTY_FUNCTION__ = "h264_encoder__encode"
#17 0x00007feb0e7ab28f in h264_encoder__do_work (handle=<optimized out>) at ../src/h264-encoder.c:419
        self = 0x163c9e0
        frame = 0x7fe994000dc0
        __PRETTY_FUNCTION__ = "h264_encoder__do_work"
        rc = <optimized out>
#18 0x00007feb0e7f87a7 in worker_fn () from /nix/store/fsfq89ri4r7rjnbh0qjdaxxj461830xn-aml-3afc3aacce6d0342bdff71c38cc477d6e23b9be3/lib/libaml.so.0
No symbol table info available.
#19 0x00007feb0e5bfe86 in start_thread () from /nix/store/fz54faknl123dimzz6jsppw193lx2mip-glibc-2.35-163/lib/libc.so.6
No symbol table info available.
#20 0x00007feb0e646c60 in clone3 () from /nix/store/fz54faknl123dimzz6jsppw193lx2mip-glibc-2.35-163/lib/libc.so.6
No symbol table info available.

@any1
Copy link
Owner

any1 commented Oct 4, 2022

plane_order is NULL, which happens if vl_video_buffer_plane_order fails to recognise the pixel format. If you can do a p buf->base.buffer_format in a debugger, we can see the pixel format.

@johnae
Copy link
Author

johnae commented Oct 4, 2022 

(gdb) p buf->base.buffer_format
$1 = PIPE_FORMAT_B8G8R8X8_UNORM

@any1
Copy link
Owner

any1 commented Oct 4, 2022

If you patch your mesa like this, does that help?

diff --git a/src/gallium/auxiliary/vl/vl_video_buffer.c b/src/gallium/auxiliary/vl/vl_video_buffer.c
index a8815c4ba77..f21ddae4abb 100644
--- a/src/gallium/auxiliary/vl/vl_video_buffer.c
+++ b/src/gallium/auxiliary/vl/vl_video_buffer.c
@@ -78,6 +78,7 @@ vl_video_buffer_plane_order(enum pipe_format format)
    case PIPE_FORMAT_NV12:
    case PIPE_FORMAT_R8G8B8A8_UNORM:
    case PIPE_FORMAT_B8G8R8A8_UNORM:
+   case PIPE_FORMAT_B8G8R8X8_UNORM:
    case PIPE_FORMAT_YUYV:
    case PIPE_FORMAT_UYVY:
    case PIPE_FORMAT_P010:

@johnae
Copy link
Author

johnae commented Oct 4, 2022 •
edited
Loading

Yeah that seems to have fixed the issue. I do see this in wlvncc though (eg. the WARNING - no idea if it's important somehow):

❯ ./wlvncc/bin/wlvncc 127.0.0.1 5900
Using EGL for rendering...
04/10/2022 16:38:10 VNC server supports protocol version 3.8 (viewer 3.8)
04/10/2022 16:38:10 We have 1 security types to read
04/10/2022 16:38:10 0) Received security type 1
04/10/2022 16:38:10 Selecting security type 1 (0/1 in the list)
04/10/2022 16:38:10 Selected Security Scheme 1
04/10/2022 16:38:10 No authentication needed
04/10/2022 16:38:10 VNC authentication succeeded
04/10/2022 16:38:10 Desktop name "WayVNC"
04/10/2022 16:38:10 Connected to VNC server, using protocol version 3.8
04/10/2022 16:38:10 VNC server default format:
04/10/2022 16:38:10   32 bits per pixel.
04/10/2022 16:38:10   Least significant byte first in each pixel.
04/10/2022 16:38:10   TRUE colour: max red 255 green 255 blue 255, shift red 16 green 8 blue 0
Oops, back-buffer is still attached.
WARNING: Long delays observed (9231). Something is blocking the main loop
WARNING: Long delays observed (27812). Something is blocking the main loop
WARNING: Long delays observed (24760). Something is blocking the main loop
WARNING: Long delays observed (14070). Something is blocking the main loop
WARNING: Long delays observed (9504). Something is blocking the main loop
... etc

So just to note what I'm running here:
I've patched mesa with the above patch but I am also running wayvnc with the 2nd patch from #159 (comment) - however I think using the first patch might work as well. Let me try that, brb.

@johnae
Copy link
Author

johnae commented Oct 4, 2022

Scratch that. I went back to your first suggested patch I remember now - so that's what I'm running.

@any1
Copy link
Owner

any1 commented Oct 4, 2022

It's just a warning.

@any1
Copy link
Owner

any1 commented Oct 4, 2022

I made a mesa PR for this: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18959

@johnae
Copy link
Author

johnae commented Oct 4, 2022

That's great! Thanks for your help.

@any1 any1 mentioned this issue Oct 6, 2022
Open
@any1
Copy link
Owner

any1 commented Oct 6, 2022

I created a new issue for fixing encodability of buffers in general, so I'm closing this.

@any1 any1 closed this as completed Oct 6, 2022
@dec05eba dec05eba mentioned this issue Feb 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@johnae @any1