ADD vaapi encoder #42

xhejtman · 2022-08-19T10:33:14Z

added vaapi encoder properly

ehfd

Looks good to me.

ehfd · 2022-08-19T10:47:13Z

Resolves part of #34.

src/selkies_gstreamer/gstwebrtc_app.py

ehfd · 2022-08-25T11:05:14Z

gst-vaapi is required to be included in the build procedures. @danisla Please keep it in mind.

danisla · 2022-09-08T20:11:08Z

@ehfd are there any changes to the gstreamer source compilation needed to get vaapi support? What are the runtime libs required?

danisla

What are the requirements to get the vaapih264enc elements? Are they currently compiled with the gstreamer base image, or are there additional compile-time or runtime dependencies?

ehfd · 2022-09-09T03:39:21Z

@danisla bundle GST-VAAPI (VA drivers required in host of course)
Also note that only i965 is supported for Intel, not iHD, and we have not tested the behaviors perfectly. We would want to potentially change to vah264enc once that is out, and perhaps amfh264enc and msdkh264enc as well.

danisla · 2022-09-09T04:36:17Z

@danisla bundle GST-VAAPI (VA drivers required in host of course) Also note that only i965 is supported for Intel, not iHD, and this is experimental

I don't have an i965 to test with, so just to confirm, no changes to the addons/gstreamer/Dockerfile are required and this is purely a runtime dependency?

ehfd · 2022-09-09T04:37:30Z

@xhejtman

ehfd · 2022-09-09T04:41:12Z

@danisla I do not see it as an available plugin on GStreamer compiled with Selkies 1.3.1.
gst-inspect-1.0 | grep vaapi

danisla · 2022-09-09T04:48:54Z

The gstreamer ubuntu20.04 build logs shows that the gstreamer-vaapi subproject is not being built.

2022-08-19T16:10:40.1095173Z Subproject gstreamer-vaapi : skipped: feature vaapi disabled

Can someone please propose and test a change to the addons/gstreamer/Dockerfile to include vaapi support so that this PR will work as expected?

ehfd · 2022-09-09T05:48:23Z

Note: set USE_DRM for gstreamer-vaapi/config.h to true. This should be somehere in the build settings.
Dependencies:

libva_req = ['>= 0.39.0', '!= 0.99.0']
libwayland_req = '>= 1.11.0'
libdrm_req = '>= 2.4.98'
gst_req = '>= @0@.@1@.0'.format(gst_version_major, gst_version_minor)

# Mandatory GST deps
libm = cc.find_library('m', required : false)
gst_dep = dependency('gstreamer-1.0', version : gst_req,
  fallback : ['gstreamer', 'gst_dep'])
gstbase_dep = dependency('gstreamer-base-1.0', version : gst_req,
  fallback : ['gstreamer', 'gst_base_dep'])
gstpbutils_dep = dependency('gstreamer-pbutils-1.0', version : gst_req,
    fallback : ['gst-plugins-base', 'pbutils_dep'])
gstallocators_dep = dependency('gstreamer-allocators-1.0', version : gst_req,
    fallback : ['gst-plugins-base', 'allocators_dep'])
gstvideo_dep = dependency('gstreamer-video-1.0', version : gst_req,
    fallback : ['gst-plugins-base', 'video_dep'])
gstcodecparsers_dep = dependency('gstreamer-codecparsers-1.0', version : gst_req,
    fallback : ['gst-plugins-bad', 'gstcodecparsers_dep'])
gstgl_dep = dependency('gstreamer-gl-1.0', version : gst_req,
    fallback : ['gst-plugins-base', 'gstgl_dep'], required: false)
gstglproto_dep = dependency('', required : false)
gstglx11_dep = dependency('', required : false)
gstglwayland_dep = dependency('', required : false)
gstglegl_dep = dependency('', required : false)

All dependencies, even not required, should also be there.

ehfd · 2022-09-09T05:48:43Z

@danisla libva, libwayland, libdrm

xhejtman · 2022-09-09T07:08:44Z

hmm, yes, to build vaapi subproject was painful. I built it manualy. Also note, for runtime, i965-va-driver-shaders is needed.

ehfd · 2022-09-09T08:15:38Z

Should we stop here for now, perhaps?
vah264enc is in main and should be in GST 1.21 gst-bad instead of a separate gst-vaapi package. It should have a better experience with minimal build process changes.

Edit: yeah for sure, let's wait on this until we see vah264enc.

xhejtman · 2022-09-09T10:17:57Z

It makes sense, it was really not easy to build it properly.

ehfd · 2022-10-13T04:50:57Z

I think this PR is ready to be revived, now that vah264enc is in 1.21.1.

ehfd · 2022-10-13T11:38:11Z

@xhejtman The vaapi issue with the framerate last time might have been with MIT-SHM missing, btw.

danisla · 2022-10-17T20:47:08Z

I've been testing with gstreamer 1.21.1, and vah264enc is present and is working with gstreamer standalone like this:

gst-launch-1.0 ximagesrc ! vapostproc ! video/x-raw,format=NV12 ! vah264enc ! fakesink

However, regardless of encoder setting, gstreamer segfaults when running with webrtcbin on gst-1.21.1. Also I am on the fence for supporting non-stable gstreamer releases (odd numbered releases) because of issues like this..,

xhejtman · 2022-10-17T22:29:09Z

which intel driver are you using? It works with shader only. Also, vapostproc is problematic. I would start either with SW/CPU video convert. And perhaps use different intel driver. iHD is mostly buggy.

danisla · 2022-10-17T22:56:14Z

@xhejtman the vaapi codecs and vapostproc seemed to work just fine, I don't think the issue was with the vaapi plugin, but rather with webrtc in gstreamer v1.21.1. Like I said, the x264enc pipeline doesn't work on 1.21.1 either.

I'm probably just going to stick with the stable releases of gstreamer since odd releases are not stable.

ehfd · 2022-10-18T00:26:12Z

Relevant? @danisla :
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/commit/430ec0d860738199b81c4aac73a6b057918be144

I think we also need to check if dependencies have changed in 276fe41.

-Dvaapi=enabled -Dvaapi:encoders=true -Dvaapi:drm=true
Moreover, is this not for the VAAPI plugin, not the VA plugin within bad?

danisla · 2022-10-18T00:27:30Z

@xhejtman I'm having trouble getting this to work with 1.20.4.

This is what my vainfo output looks like:

libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva error: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva info: va_openDriver() returns 1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Bay Trail - 2.4.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264StereoHigh         : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileJPEGBaseline           : VAEntrypointVLD

then gst-inspect-1.0 vaapi shows this:

0:00:00.656034240    25 0x564effb1a200 WARN                 default gstvaapi.c:231:plugin_init: Cannot create a VA display
Plugin Details:
  Name                     vaapi
  Description              VA-API based elements
  Filename                 /opt/gstreamer/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
  Version                  1.20.4
  License                  LGPL
  Source module            gstreamer-vaapi
  Source release date      2022-10-12
  Binary package           gstreamer-vaapi
  Origin URL               Unknown package origin


  0 features:

ehfd · 2022-10-18T00:32:38Z

@danisla First thing. Is /dev/dri/cardX from an Intel GPU accessible?

ehfd · 2022-10-18T00:36:43Z

https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/

danisla · 2022-10-18T00:38:25Z

@danisla First thing. Is /dev/dri/cardX from an Intel GPU accessible?

yes, as you can see from my previous comment, the vainfo output is good, permissions are good on /dev/dri.

ehfd · 2022-10-18T00:43:01Z

I'll try to fiddle with the build process now. Though I don't have a Linux node with Intel right now.

ehfd · 2022-10-18T00:53:43Z

gst1-21-1.txt
What I see so far. Without -Dvaapi=enabled -Dvaapi:encoders=true -Dvaapi:drm=true so all these have nothing to do with va in gst-bad.

danisla · 2022-10-18T01:05:08Z

I would like to see this working with 1.20.4 if possible. I am assuming that is what @xhejtman got it working with.

We control the gstreamer build so we should be able to make it work if we know the build steps.

ehfd · 2022-10-18T01:10:19Z

I also see a segfault on 1.21.1.

I believe it might be related to https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1508, although not limited to NVIDIA

danisla · 2022-10-18T01:19:02Z

I also see a segfault on 1.21.1.

Edit: It happens right here

0:00:34.057128636 45609      0x24cc980 DEBUG               GST_PADS gstpad.c:4193:gst_pad_query:<capsfilter2:sink> sent query 0x1942cf0 (caps), result 1
0:00:34.057139942 45609      0x24cc980 DEBUG               GST_CAPS gstpad.c:2837:gst_pad_get_allowed_caps:<nvenc:src> allowed caps video/x-h264, width=(int)[ 33, 4096 ], height=(int)[ 17, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high
0:00:34.064355940 45609      0x24cc980 INFO                   nvenc gstnvbaseenc.c:1864:gst_nv_base_enc_set_format:<nvenc> configured encoder
0:00:34.064378957 45609      0x24cc980 DEBUG                  nvenc gstnvbaseenc.c:1624:gst_nv_base_enc_calculate_num_prealloc_buffers:<nvenc> Calculated num buffers: 5 (lookahead 0, frameIntervalP 1)
0:00:34.067076449 45609      0x24cc980 INFO                   nvenc gstnvbaseenc.c:1959:gst_nv_base_enc_set_format:<nvenc> allocated output buffer  0: 0x7f708008fe10
0:00:34.068826456 45609      0x24cc980 INFO                   nvenc gstnvbaseenc.c:1959:gst_nv_base_enc_set_format:<nvenc> allocated output buffer  1: 0x7f7080005550
0:00:34.070164906 45609      0x24cc980 INFO                   nvenc gstnvbaseenc.c:1959:gst_nv_base_enc_set_format:<nvenc> allocated output buffer  2: 0x7f70804e8180
0:00:34.071452229 45609      0x24cc980 INFO                   nvenc gstnvbaseenc.c:1959:gst_nv_base_enc_set_format:<nvenc> allocated output buffer  3: 0x7f708017a610
/etc/selkies-gstreamer-entrypoint.sh: line 57: 45609 Segmentation fault      (core dumped) selkies-gstreamer --json_config="${SELKIES_USER_CONFIG_FILE}" --addr="0.0.0.0" --port="8080" $@

I believe it might be https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1508.

I don't think it's related to the encoder, it segfaults for me with any encoder.

See my previous comment:

However, regardless of encoder setting, gstreamer segfaults when running with webrtcbin on gst-1.21.1. Also I am on the fence for supporting non-stable gstreamer releases (odd numbered releases) because of issues like this..,

ehfd · 2022-10-18T01:25:53Z

However, regardless of encoder setting, gstreamer segfaults when running with webrtcbin on gst-1.21.1. Also I am on the fence for supporting non-stable gstreamer releases (odd numbered releases) because of issues like this..,

I understand the odd numbered releases are unreliable, but I do still need to determine if it's on our side or it's the code. On second look, it happens right when the SDP connection is established on any encoder, yes.

ehfd · 2022-10-18T01:33:27Z

@danisla
Strangely, x264 and vp8 works with gst 1.21.1 after I did something outside of my understanding.
Alhough logs are spamming with 0:01:04.799529049 77363 0x7f716c0a24c0 WARN rtpsession gstrtpsession.c:2438:gst_rtp_session_chain_send_rtp_common:<rtpsession0> Can't determine running time for this packet without knowing configured latency

Let me refresh my container and setup again.

Edit: I definitely did not confuse versions. They work in a certain condition, but is inconsistent... Need to find what happened.

ehfd · 2022-10-18T02:58:50Z

Overall, however, the behavior of 1.21 is too puzzling to be used for Selkies.

xhejtman · 2022-10-18T07:06:45Z

Do you have i965-va-driver or i965-va-driver-shaders?

ehfd · 2022-10-18T10:58:26Z

@xhejtman Check https://github.com/selkies-project/selkies-gstreamer/blob/gst-1.21.1/Dockerfile.example and https://github.com/selkies-project/selkies-gstreamer/blob/gst-1.21.1/addons/gstreamer/Dockerfile.

Overall: 276fe41
Do you mean we need them while building GST? I didn't think so.

xhejtman · 2022-10-18T11:02:54Z

so, it looks like you have the shaders version. strange, so it might be related to particular intel gpu, some might work, some might not.

danisla · 2022-10-19T15:40:53Z

I got it working with gstreamer 1.20, was missing a build time dependency of libudev-dev

I'll create a new PR to update to the gstreamer 1.20 branch with vaapi support.

ehfd · 2022-10-19T16:17:18Z

I got it working with gstreamer 1.20, was missing a build time dependency of libudev-dev

I'll create a new PR to update to the gstreamer 1.20 branch with vaapi support.

Does it happen to work with the VA plugin (not VAAPI)? It probably needs a plugin rank change if in 1.20.

danisla · 2022-10-19T16:19:43Z

Only vaapi, the va plugin does not have the encoder elements in gstreamer 1.20, we'll have to wait until 1.22 to support the new va plugin.

ehfd · 2022-10-19T16:37:51Z

Please change the docs too if there are runtime dependency changes.
There should be 2 places that should be changed.

ehfd · 2022-11-26T20:26:10Z

@danisla @xhejtman

Issue with Selkies on GST 1.21.x raised on https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1605

ehfd · 2023-08-17T13:46:57Z

Implemented in 736292c

ADD vaapi encoder

d12de8b

ehfd approved these changes Aug 19, 2022

View reviewed changes

ehfd mentioned this pull request Aug 19, 2022

Update gstwebrtc_app.py #41

Closed

danisla requested changes Aug 24, 2022

View reviewed changes

src/selkies_gstreamer/gstwebrtc_app.py Outdated Show resolved Hide resolved

src/selkies_gstreamer/gstwebrtc_app.py Outdated Show resolved Hide resolved

replace convert with vaapipostproc

2451b90

ehfd mentioned this pull request Aug 25, 2022

[META] Codec improvement and additional support including AV1/H.265, Intel/AMD GPU encoding support #34

Open

danisla self-requested a review September 8, 2022 20:07

danisla reviewed Sep 8, 2022

View reviewed changes

ehfd self-requested a review October 13, 2022 11:37

ehfd removed their request for review October 17, 2022 23:54

ehfd self-requested a review November 27, 2022 13:07

ehfd closed this Aug 17, 2023

ADD vaapi encoder #42

ADD vaapi encoder #42

Conversation

xhejtman commented Aug 19, 2022

ehfd left a comment

Choose a reason for hiding this comment

ehfd commented Aug 19, 2022

ehfd commented Aug 25, 2022

danisla commented Sep 8, 2022

danisla left a comment

Choose a reason for hiding this comment

ehfd commented Sep 9, 2022 • edited

danisla commented Sep 9, 2022

ehfd commented Sep 9, 2022

ehfd commented Sep 9, 2022 • edited

danisla commented Sep 9, 2022 • edited

ehfd commented Sep 9, 2022 • edited

ehfd commented Sep 9, 2022

xhejtman commented Sep 9, 2022

ehfd commented Sep 9, 2022 • edited

xhejtman commented Sep 9, 2022

ehfd commented Oct 13, 2022

ehfd commented Oct 13, 2022

danisla commented Oct 17, 2022 • edited

xhejtman commented Oct 17, 2022

danisla commented Oct 17, 2022

ehfd commented Oct 18, 2022 • edited

danisla commented Oct 18, 2022

ehfd commented Oct 18, 2022

ehfd commented Oct 18, 2022

danisla commented Oct 18, 2022

ehfd commented Oct 18, 2022

ehfd commented Oct 18, 2022 • edited

danisla commented Oct 18, 2022

ehfd commented Oct 18, 2022 • edited

danisla commented Oct 18, 2022 • edited

ehfd commented Oct 18, 2022

ehfd commented Oct 18, 2022 • edited

ehfd commented Oct 18, 2022

xhejtman commented Oct 18, 2022

ehfd commented Oct 18, 2022 • edited

xhejtman commented Oct 18, 2022

danisla commented Oct 19, 2022

ehfd commented Oct 19, 2022

danisla commented Oct 19, 2022

ehfd commented Oct 19, 2022 • edited

ehfd commented Nov 26, 2022

ehfd commented Aug 17, 2023

ehfd commented Sep 9, 2022 •

edited

ehfd commented Sep 9, 2022 •

edited

danisla commented Sep 9, 2022 •

edited

ehfd commented Sep 9, 2022 •

edited

ehfd commented Sep 9, 2022 •

edited

danisla commented Oct 17, 2022 •

edited

ehfd commented Oct 18, 2022 •

edited

ehfd commented Oct 18, 2022 •

edited

ehfd commented Oct 18, 2022 •

edited

danisla commented Oct 18, 2022 •

edited

ehfd commented Oct 18, 2022 •

edited

ehfd commented Oct 18, 2022 •

edited

ehfd commented Oct 19, 2022 •

edited