Render for websites [Proof of Concept,Working]

[josephm1]

Hi Jeff, I am writing to request that you consider adding a web renderer for Hanabi as a feature in your extension. I have included a working example of the web renderer. As a newcomer to Gnome JavaScript, I am not familiar with the best practices for integrating this feature into the rest of your extension. I appreciate the excellent work you have done so far and hope that you will consider adding this feature. If you have any questions let me know. Thanks and best wishes!

[jeffshee]

Hi @josephm1,
Thanks for your contribution! I appreciate it 😊

Based on my experience with Hidamari (another live wallpaper app) though, I don't think that using WebKitGTK is a good choice here. Its graphic implementation is totally unusable.
For example, if you try Epiphany with websites like shadertoy, you will see most of them don't work, are laggy, or even crash.
https://flathub.org/apps/details/org.gnome.Epiphany

There are 2 alternative solutions which I haven't explored much yet:

• Write a renderer with Chromium Embedded Framework.
  (Actually, it doesn't need to be a GTK app. I picked gjs simply because it's included in a standard GNOME desktop environment.)
• Use installed browsers (like Firefox/Chrome) of users' system
  (Only viable if we have enough control over the browsers by using their CLI flags/arguments, such that it's possible to control the window's title and size, toggle fullscreen mode, etc.)

Please tell me if you have any ideas~ Again I appreciate your contribution!

[Rafostar]

Write a renderer with Chromium Embedded Framework.
(Actually, it doesn't need to be a GTK app. I picked gjs simply because it's included in a standard GNOME desktop environment.)

There is a GStreamer element that is supposed to do that (haven't tried it myself): https://github.com/centricular/gstcefsrc.

With it installed on the system, you should be able to simply set URI (like you normally do in your current code) to web+https://example.com and it "might" just work without any changes to hanabi itself.

[josephm1]

Hi @jeffshee @Rafostar , I tried the chromium embedded framework and different alternatives like trying to get a headless chrome/Firefox to stream its content to display, but I was not able to get anything to work reliably and to be performant.

I know gnome web (Epiphany) has not been reliable or performant in the past, however it has improved considerably, especially since its recent updates. I just tested it there it appears to be comparable in speed to chrome on Linux, currently using Shader Toy as a benchmark. I am just wondering if you think it might be suitable to use now?

[Rafostar]

@josephm1
Out of curiosity, have you tried the gstreamer plugin I linked previously?

[josephm1]

It was a while ago so I'm not 100% certain. I believe I was not able to get Gstreamer plugin to work at all but I had some success with just getting websites to display using Spotify's CEF but it was not clear to me how to link to GTK and then I tried the python bindings (which are no longer maintained) which often crashed on my laptop any time it did manage to work.

Again though, I wouldn't have had any experience with these technologies before so there may have been something simple I was not doing.

[Rafostar]

Works with xvimagesink for me. I even played youtube video by rendering website into window. GstNavigation does not seem to be implemented yet, so someone would need to contribute to add support for it probably.

[jeffshee]

Works with xvimagesink for me as well, even with GPU option enabled, and shadertoy works too.
The plugins directory used in command below contains both gstcefsrc and gst-plugin-gtk4 plugins.

GST_PLUGIN_PATH="plugins:$GST_PLUGIN_PATH" gst-launch-1.0 cefbin name=cef cefsrc::url="https://www.shadertoy.com/view/WtXcWB" cefsrc::gpu=true ! video/x-raw, width=1920, height=1080, framerate=60/1 ! videoconvert ! xvimagesink cef.audio ! audioconvert ! audiomixer ! autoaudiosink

Also, gtk4paintablesink seems working. (I haven't tested it with GtkImage, just with gst-launch and the pipeline set to playing state successfully)

GST_PLUGIN_PATH="plugins:$GST_PLUGIN_PATH" gst-launch-1.0 cefbin name=cef cefsrc::url="https://www.shadertoy.com/view/WtXcWB" cefsrc::gpu=true ! video/x-raw, width=1920, height=1080, framerate=60/1 ! videoconvert ! gtk4paintablesink cef.audio ! audioconvert ! audiomixer ! autoaudiosink

@Rafostar No luck with clappersink tho. For some reason, the pipeline stuck in the paused state. Please tell me if I had done anything wrong...

GST_PLUGIN_PATH="plugins:$GST_PLUGIN_PATH" gst-launch-1.0 cefbin name=cef cefsrc::url="https://www.shadertoy.com/view/WtXcWB" cefsrc::gpu=true ! video/x-raw, width=1920, height=1080, framerate=60/1 ! videoconvert ! clappersink cef.audio ! audioconvert ! audiomixer ! autoaudiosink

@josephm1 I don't have much faith for Epiphany, at least atm. I have the latest technology preview (flatpak) installed on my system, the webpage does load, but it doesn't show anything (not even Google homepage lol!). Well, I think it's mostly because I have an Nvidia GPU... 😔

EDIT: Mistake in first command

[josephm1]

Thank you @jeffshee and @Rafostar , is there anything I could do to help get this to be implemented in hanabi?

[jeffshee]

@josephm1, how about implementing the Preferences Dialog (prefs.js)? It will be essential for users to set the URL or choose the local HTML file later on. If you're up for it, I'll handle the gstcefsrc integration.

We can even discuss this on Discord if that's easier to have a conversation. (Maybe a Discord server so that everyone can exchange opinion...?) Let me know your thoughts. 😄

[Rafostar]

@jeffshee

No luck with clappersink tho.

Theres probably some deadlock. Will probably need to investigate with gdb this.

I'll handle the gstcefsrc integration.

What integration you are speaking about? All you need is some dialog to enter to URL (web+https://...). Then just set it to GstPlay you are already using and it should work with your current renderer code.

[jeffshee]

@Rafostar

Will probably need to investigate with gdb this.

I don't have any experience with gdb. But please tell me if there is anything I could help anyway.

What integration you are speaking about?

Will try to add codes to meson that build the plugins from sources, as these plugins aren't available out of the box. Probably will also need to configure GstRegistry so that it loads those plugins during runtime.

All you need is some dialog to enter to URL (web+https://...). Then just set it to GstPlay you are already using and it should work with your current renderer code.

Hmm, is it that straightforward? I thought I have to do something with the pipeline. gstcefsrc seems to be capped at 30fps 720p and use gpu=false as default, and I still haven't figured out a way to change those parameters with GstPlay. Is it even possible to do so in GstPlay?

[Rafostar]

Hmm, is it that straightforward? I thought I have to do something with the pipeline. gstcefsrc seems to be capped at 30fps 720p and use gpu=false as default, and I still haven't figured out a way to change those parameters with GstPlay. Is it even possible to do so in GstPlay?

Ah, then probably not from GJS. It will be tricky to change them from GstPlay, but even if you build a manual pipeline I imagine you will run into all kinds of issues when you will try to control it by connecting to its signals. Like signals not being emitted or some other deadlocks or crashes.