Performance issues

[VarLad]

Currently enabling this, the gjs process takes about 9-11% of my CPU.
Any way to use hardware acceleration here?

[jeffshee]

Hi @VarLad, thanks for reporting. I'm not an expert to this actually, so I might be wrong.

It depends on what kind of GPU you have. The backend used here is gstreamer.
First, try to get some info with gst-inspect. If you are using NVidia card, this should be expected output:

❯ gst-inspect-1.0 nvcodec
Plugin Details:
  Name                     nvcodec
  Description              GStreamer NVCODEC plugin
  Filename                 /usr/lib64/gstreamer-1.0/libgstnvcodec.so
  Version                  1.20.3
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2022-06-15
  Binary package           Fedora GStreamer-plugins-bad package
  Origin URL               http://download.fedoraproject.org

  cudadownload: CUDA downloader
  cudaupload: CUDA uploader
  nvh264dec: NVDEC h264 Video Decoder
  nvh264enc: NVENC H.264 Video Encoder
  nvh264sldec: NVDEC H.264 Stateless Decoder
  nvh265dec: NVDEC h265 Video Decoder
  nvh265enc: NVENC HEVC Video Encoder
  nvh265sldec: NVDEC H.265 Stateless Decoder
  nvjpegdec: NVDEC jpeg Video Decoder
  nvmpeg2videodec: NVDEC mpeg2video Video Decoder
  nvmpeg4videodec: NVDEC mpeg4video Video Decoder
  nvmpegvideodec: NVDEC mpegvideo Video Decoder
  nvvp8dec: NVDEC vp8 Video Decoder
  nvvp8sldec: NVDEC VP8 Stateless Decoder
  nvvp9dec: NVDEC vp9 Video Decoder
  nvvp9sldec: NVDEC VP9 Stateless Decoder

  16 features:
  +-- 16 elements

If you are using Intel/AMD, I guess this should be the expected one:

❯ gst-inspect-1.0 vaapi         
Plugin Details:
  Name                     vaapi
  Description              VA-API based elements
  Filename                 /usr/lib64/gstreamer-1.0/libgstvaapi.so
  Version                  1.20.3
  License                  LGPL
  Source module            gstreamer-vaapi
  Source release date      2022-06-15
  Binary package           gstreamer-vaapi
  Origin URL               Unknown package origin

  vaapidecodebin: VA-API Decode Bin
  vaapih263dec: VA-API H263 decoder
  vaapih264dec: VA-API H264 decoder
  vaapimpeg2dec: VA-API MPEG2 decoder
  vaapimpeg4dec: VA-API MPEG4 decoder
  vaapipostproc: VA-API video postprocessing
  vaapisink: VA-API sink
  vaapivc1dec: VA-API VC1 decoder

  8 features:
  +-- 8 elements

From gstreamer documentation, it says

Sneak peek: if properly setup, you do not need to do anything special to activate hardware acceleration; GStreamer automatically takes advantage of it.

So, if the process has a high CPU usage, probably there is something wrong with your setup. Or the video you are trying to play is not HW acceleration supported by gstreamer.

If your GPU use VA-API (Intel/AMD), then you should also check vainfo and see if there is any entrypoint.

Some articles that might be helpful:
https://wiki.archlinux.org/title/Hardware_video_acceleration
https://lifestyletransfer.com/how-to-install-gstreamer-vaapi-plugins-on-ubuntu/

[jeffshee]

That being said, it seems that I also couldn't get the HW acceleration working on my side. (High CPU usage when compare to other video player, for example Clapper which is also gstreamer-based AFAIK)...

Need some investigation. Any advice is welcome!

[VarLad]

@jeffshee I think for gstreamer plugin for vaapi we have gstreamer1-vaapi package on Fedora
I'm not near my machine right now, but do tell if installing that makes a difference for you

[jeffshee]

@VarLad This extension doesn't use anything from Flathub. It is not a Flatpak. So the HW acceleration setup on host must be done correctly.

[VarLad]

@jeffshee Sorry, I mistyped, but I quickly edited the issue :3
Again, I'd recommend to see if installinggstreamer1-vaapi package in Fedora makes a difference.

[jeffshee]

@VarLad

Again, I'd recommend to see if installing gstreamer1-vaapi package in Fedora makes a difference.

Yep, we should experiment with this first. Did installing gstreamer1-vaapi work for you? (It doesn't work for me tho)

[VarLad]

Didn't work for me either :<
On the other hand, @jeffshee I found https://gstreamer.freedesktop.org/documentation/tutorials/playback/hardware-accelerated-video-decoding.html?gi-language=c
and I remember that Clapper too had different ranks for hw-decoding too like this
Can you try setting the va plugin in gstreamer (should handle most formats)?

[Rafostar]

Hi, somebody pinged me about this problem for help. The easiest way to check what your app exactly uses is to generate a pipeline dot file to some directory GST_DEBUG_DUMP_DOT_DIR=/some/dir.

AFAIU from quick glance at your code, you use GTK media stream to play video in your app and set it as Shell background, right? GTK media stream is a very simple and basic example for testing video playback within GTK4, but in order to maximize performance and reduce CPU usage as much as possible, what you really should do is use custom GStreamer pipeline instead. This way you can for example not software decode audio at all instead just muting it, etc.

[VarLad]

The creator of Clapper themselves :D

@Rafostar Any idea why hardware decoding is not working here?

[jeffshee]

@Rafostar, wow thanks for the advice! (I'm a Clapper user too =3)

TBH, Gstreamer stuff is currently outside of my expertise. I managed to generate the dot files but have no idea what info I should look for. I attached mine here if you don't mind taking a look =)
pipeline.zip

GTK media stream is a very simple and basic example for testing video playback within GTK4, but in order to maximize performance and reduce CPU usage as much as possible, what you really should do is use custom GStreamer pipeline instead.

It's indeed very sad to know that GTK4's implementation sometimes lacks this and that. Is that mean I have to follow the path that Clapper is doing (Gstreamer -> OpenGL -> Gtk4)? How much work needs to be done, if all I need is just a simple window playing video in a loop, with HW acceleration? Any documentation suggestions?

[Rafostar]

@jeffshee
Thanks for the dot files. As I suspected, it IS using hardware acceleration for decoding 😄. The example you send was using VP9 hw accel with Nvidia binary driver (nvvp9dec element).

The problem here is that it does not go with GLMemory but raw frames. Basically it goes like this: frame is hw decoded, then it is downloaded from GPU memory to system memory (which already puts pressure on system memory bandwidth), later software colorspace conversion happens and its uploaded back to GPU within GTK4 so it can be shown. All this results in high CPU usage that you see.

There is no much control from user side over GTK media stream implementation "auto" behaviour as it doesn't have access to underlying GStreamer pipeline at all (BTW: I do not use it in Clapper either). But do you even need a GUI toolkit for anything here? AFAIK you just want to spawn a process that shows video only, no?

[jeffshee]

@Rafostar
Thanks for the detailed explanation, that makes a lot of sense! 😄

But do you even need a GUI toolkit for anything here? AFAIK you just want to spawn a process that shows video only, no?

Yep, probably a process that allows some basic control over its spawned window, is enough. For the extension to work, the window needs to have a magic string @io.github.jeffshee.hanabi_renderer!0,0;H as its title. Some basic window operations such as minimize, full-screen, and disable window decoration, are also desirable.

[Rafostar]

Some basic window operations such as minimize, full-screen, and disable window decoration, are also desirable.

@jeffshee
That's quite a few features for a wallpaper 😆. For something easy to implement and universal (works on both xorg, wayland) without much code changes, one way would be using for e.g. GstPlay API + some video sink. There is video overlay renderer iface in there that uses window handle, but AFAIK GJS does not have bindings for this functionality AND this is not working with Wayland.

An easy to do replacement here would be using GstPlay API + clapper sink, but this would introduce a Clapper dependency to your extension, something that you probably do not want doing. Other then that, you could investigate why GtkMedia sink decides to use raw frames. But as I mentioned earlier, its very limited and (unlike GstPlay) it doesn't give you option to not decode audio.

[jeffshee]

@Rafostar
Thanks! I think I should go through the Gstreamer docs and look at Clapper's codes to understand better how things work under the hood. I might ask you some questions in Clapper's Matrix if you don't mind! =3

An easy to do replacement here would be using GstPlay API + clapper sink, but this would introduce a Clapper dependency to your extension, something that you probably do not want doing.

I'm not against the idea of ​​introducing a Clapper dependency. If the player stuff can't fit in a simple GJS script, I would rather separate them from the extension part and package them into Flatpaks~ Users will have to install 2 separate components tho, which is... not too bad IMO.

[Rafostar]

@jeffshee
If you are not against option of using Clapper for improved performance, then my suggestion to make this universal would be: do a little code refactor and use Clapper sink when installed otherwise fallback to current GtkMediaStream implementation (your current code).

I can help you with this by doing a MR, so you won't have to reverse engineer how some parts in Clapper work 😄. Just let me know. I do work on some easy-to-use Clapper player API, its not ready yet, but since Clapper 0.5 the sink alone is installed and publicly available as first step. For now can be used with GStreamer provided GstPlay API which should be more then enough for your use-case here (looping a video), and maybe even your other project if you decide to port it to GTK4: jeffshee/hidamari#64 instead of bundling VLC there.

Remember, that this is only my suggestion. I do not want to put a pressure/force you into this if you do not want to.

[jeffshee]

@Rafostar

I can help you with this by doing a MR, so you won't have to reverse engineer how some parts in Clapper work smile. Just let me know.

That sounds great! I would appreciate it if you could help me out with Gstreamer stuff! 😆

About the Clapper player API that you mentioned, will that be a library like libvlc or libmpv, but for Gtk4? If that is the case, I think it will be an awesome addon to the Gtk4 itself!

[Rafostar]

@jeffshee

About the Clapper player API that you mentioned, will that be a library like libvlc or libmpv, but for Gtk4?

Something like GstPlay API that GStreamer offers but easier to use for people not familiar with GStreamer and hopefully not as buggy. Right now I am using modified version of their older API in Clapper but I want to write and switch into using my own that is easier to understand, has more cool features and works better with (but not limited to) GTK4. Feel free to ask any Clapper related stuff on Clapper matrix channel 😄