OpenGL and Wayland

[brinko99]

We're really struggling to get any OpenGL rendering with Uno (4.10.13) and Skia on our custom Linux-embedded image. We would appreciate any help from those who have gone down this path. We're finding that basic UI performance is lacking with software rendering.

Like others (#8643), the app bombs out with either OpenGL or OpenGLES rendering options.

trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Loaded eagerly: /firmware/user/tlouser/LocalUI/libHarfBuzzSharp.so
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Could not load eagerly: libHarfBuzzSharp
Gdk-Message: 19:22:29.965: Unable to load sb_up_arrow from the cursor theme
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
Marshaling realize signal
GLIB UNHANDLED EXCEPTIONSystem.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Uno.UI.Runtime.Skia.GLRenderSurfaceBase.GLRenderSurface_Realized

The common workaround appears to be to run under X11 (GDK_BACKEND=x11) using Xwayland. We've tried that to no avail: Gtk-WARNING **: cannot open display: . We've tried various DISPLAY settings, all with similar results. We are setting GDK_GL=gles.

We're using an NXP i.MX8-X ARM Cortex-A35 which should support both OpenGL and GL ES. As far as we can tell, the requisite packages are installed; demo apps such as glmark2-es2-wayland run, demonstrating the hardware rendering.

We've done some troubleshooting by looking at Uno's interaction with SkiaSharp. When we call GRGlInterface.CreateGles() with Native Context "libGLESv2.so", we can see that TryGetProcAddress gets called once with glGetString returning a reasonable looking addr, yet CreateGles returns null.

var glInterface = GRGlInterface.CreateGles(proc =>
{
    if (nativeContext.TryGetProcAddress(proc, out var addr))
    {
        return addr;
    }

    return IntPtr.Zero;
});

I'm a bit surprised the lambda gets called only once. It appears that several queries of the library are typical:
How do I properly build a .NET 7 application using SkiaSharp on Linux?.

We see the same behavior when calling GRGlInterface.Create() with Native Context "libGL.so.1". Our feeling that our Linux image is likely missing something; but we've run out of ideas.

=======================================================
    OpenGL Information
    GL_VENDOR:     Vivante Corporation
    GL_RENDERER:   Vivante GC7000L
    GL_VERSION:    OpenGL ES 3.1 V6.4.3.p4.398061
=======================================================

$ ll /usr/lib/libG*
  -rw-r--r-- 1 root root 1.8M Apr  5  2011 /usr/lib/libGAL.so
  lrwxrwxrwx 1 root root   14 Apr  5  2011 /usr/lib/libGL.so.1 -> libGL.so.1.2.0
  -rw-r--r-- 1 root root 3.5M Apr  5  2011 /usr/lib/libGL.so.1.2.0
  lrwxrwxrwx 1 root root   18 Apr  5  2011 /usr/lib/libGLESv2.so -> libGLESv2.so.2.0.0
  lrwxrwxrwx 1 root root   18 Apr  5  2011 /usr/lib/libGLESv2.so.2 -> libGLESv2.so.2.0.0
  -rw-r--r-- 1 root root 1.6M Apr  5  2011 /usr/lib/libGLESv2.so.2.0.0
  -rw-r--r-- 1 root root 1.2M Apr  5  2011 /usr/lib/libGLSLC.so

Again, any ideas are appreciated. Thanks

[jeromelaban]

Thanks for the report! Indeed, the null reference is not supposed to happen and it generally failed way before that point. We'll add some safeguards to avoid this.

That being said, we're not aware of specifics that could cause this particular issue, which could be related to GTK itself. It looks similar to GtkSharp/GtkSharp#292.

Could you try running a simple GLArea sample on your machine, directly in C++, so that we can determine if the issue comes from Gtk?

[brinko99]

We've been able to test our config with the gtk3-demo app.

All of the basic drawing demos work, but indeed the "OpenGL Area" test fails with "No GL Implementation is available". It does seem like there's something missing from our config.

If anyone comes across this and has thoughts, ideas would be appreciated.

[HakanL]

I was able to run gtk3-demo on my RPi/CM4 Wayland instance as well, including the OpenGL Area by starting it like this: GDK_GL=gles gtk3-demo. That being said, I haven't been able to get my Uno app to run accelerated (or so I assume, I haven't been able to add logging for the early startup, but setting SurfaceType to GL or GLES throws exception).

[HakanL]

I've made some progress with my set up (which I believe is related). I'm able to run Uno with OpenGL rendering, but OpenGLES fails. FWIW I'm running my set up in two docker containers, one for the wayland server, and one for my application. For the longest time I was struggling to allow the user (root) in my application to access the xwayland server, got this error:
Authorization required, but no authorization protocol specified
But I finally managed to work around that my executing this on the wayland server: xhost +SI:localuser:root.

After that I was able to run glxinfo -B:

name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Broadcom (0x14e4)
    Device: V3D 4.2 (0xffffffff)
    Version: 20.3.5
    Accelerated: yes
    Video memory: 3845MB
    Unified memory: yes
    Preferred profile: compat (0x2)
    Max core profile version: 0.0
    Max compat profile version: 2.1
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.1
OpenGL vendor string: Broadcom
OpenGL renderer string: V3D 4.2
OpenGL version string: 2.1 Mesa 20.3.5
OpenGL shading language version string: 1.20

OpenGL ES profile version string: OpenGL ES 3.1 Mesa 20.3.5
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10

And turning on early Uno trace logging shows this:
GDK_BACKEND=x11 dotnet DMXCore100.Skia.Gtk.dll

trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Loaded eagerly: /root/VSLinuxDbg/DMXCore100.Linux/runtimes/linux-arm64/native/libHarfBuzzSharp.so
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Could not load eagerly: libHarfBuzzSharp
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      OnWindowStateChanged: Focused/Withdrawn, Focused
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      OnWindowStateChanged: 0/Focused
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      GLValidationSurface: UseEs=False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL ES conditions: isMacOS=False isWSL2=False isGDKGL_GLES=False
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      OpenGL ES cannot be used (OpenGL ES is not available on this platform (Context creation failed))
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      Validating OpenGL Extensions
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      OnWindowStateChanged: Maximized, Focused/Maximized, Focused
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      GLValidationSurface: UseEs=False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL ES conditions: isMacOS=False isWSL2=False isGDKGL_GLES=False
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      OpenGL ES cannot be used (OpenGL ES is not available on this platform (Context creation failed))
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      Validating OpenGL Extensions
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Dispatch Iteration
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      Auto-detecting surface type
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Dispatch Iteration
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      Auto-detected OpenGL rendering
info: Uno.UI.Runtime.Skia.GtkHost[0]
      Using OpenGL rendering

But with GLES (and Wayland) I get this:
GDK_GL=gles GDK_BACKEND=wayland dotnet DMXCore100.Skia.Gtk.dll

trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Loaded eagerly: /root/VSLinuxDbg/DMXCore100.Linux/runtimes/linux-arm64/native/libHarfBuzzSharp.so
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Could not load eagerly: libHarfBuzzSharp
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      OnWindowStateChanged: 0/Withdrawn
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      GLValidationSurface: UseEs=False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL ES conditions: isMacOS=False isWSL2=False isGDKGL_GLES=True
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      OpenGL ES cannot be used (OpenGL ES is not available on this platform (Context creation failed))
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      Validating OpenGL Extensions
fail: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support on this system is missing extension "GL_ARB_vertex_array_object", you may need to enable software rendering. (https://platform.uno/docs/articles/features/using-skia-gtk.html#changing-the-rendering-target)
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      OpenGL cannot be used (OpenGL is not supported in this system)
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Dispatch Iteration
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      Auto-detecting surface type
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      OnWindowStateChanged: Maximized, Focused/Maximized, Focused
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      GLValidationSurface: UseEs=False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL ES conditions: isMacOS=False isWSL2=False isGDKGL_GLES=True
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      OpenGL ES cannot be used (OpenGL ES is not available on this platform (Context creation failed))
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support: isAvailable:True isMacOs:False
dbug: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      Validating OpenGL Extensions
fail: Uno.UI.Runtime.Skia.OpenGLESRenderSurface[0]
      OpenGL support on this system is missing extension "GL_ARB_vertex_array_object", you may need to enable software rendering. (https://platform.uno/docs/articles/features/using-skia-gtk.html#changing-the-rendering-target)
dbug: Uno.UI.Runtime.Skia.GLValidationSurface[0]
      OpenGL cannot be used (OpenGL is not supported in this system)
trce: Uno.UI.Runtime.Skia.GtkHost[0]
      Dispatch Iteration
dbug: Uno.UI.Runtime.Skia.GtkHost[0]
      Auto-detected Software rendering
info: Uno.UI.Runtime.Skia.GtkHost[0]
      Using Software rendering

Which could be an issue with Wayland vs X11? Any time I set GDK_BACKEND=wayland I get software rendering. If I combine GDK_GL=gles with GDK_BACKEND=x11 it detects it as OpenGL but I'm just getting black output on my display.

I'm sorry if I've hijacked your discussion @brinko99, but it seems we're trying to solve a similar problem, although on different platforms.

My main questions now are:

• What is the minimum GLES version that is required for Uno? I have 3.1 and the GL_ARB_vertex_array_object extension is listed by glxinfo, but it's still reported as missing
• Are there any benefits to use GLES over GL?
• Is it possible to run Uno with GL/GLES natively on Wayland (so I don't need to run Xwayland)? I can run gtk3-demo on Wayland, but Uno seems to require the backend to be X11.

[brinko99]

@HakanL, it looks like you've managed to get further than we. May I ask some specific questions? I am not our Linux guy unfortunately and embedded Linux is new to us generally.

As mentioned, it appears we're missing some fundamentals of GL/GLES on our system. The OpenGL Area example shows "No GL Implementation is available", even with GDK_GL=gles or GDK_GL=gl. Including GDK_BACKEND=x11 returns "Cannot open display:" even though we're supposed to have XWayland.

If you're using Yocto, what relevant Yocto recipes are you using?

You mentioned glxinfo. I was told this is not available to us as it's only X11. Is that available under Wayland or only XWayland?

As for your specific issue, in case you missed it, there was a discussion of the GL_ARB_vertex_array_object here:
#11365. The TryValidateExtensions code to validate GLES extensions is a bit suspect. I've had luck copying this source to our own codebase and stepping through.

[HakanL]

@brinko99 I'm using Yocto indirectly, I'm using the Balena.io platform and their RPi4/CM4 image (that is built using Yocto). Their platform is set up to run a lightweight docker so I have two docker instances, one for wayland and one for my application. I struggled for a while to get Xwayland to be available to my application container (issue with xhost and my lack of X authentication knowledge), and it seems that X is required to get things wired up, including for glxinfo and Uno (so yes Xwayland seems to be required).

Yeah, I saw that thread about GL_ARB_vertex_array_object, I haven't gone down the path of building the Uno source. It would be good to know if the extension/GLES library fails to load, or if it's just not finding the correct extension (which glxinfo reports as available). Maybe that's the next step, but it would be good to know if GLES is worth the extra effort. Right now I have some stability issues with GL, the display output just stops after a few minutes (works great in Framebuffer mode). Also the UI performance I get in my app isn't great, I was hoping that the app would be snappier with GL than software rendering in FB, questionable improvement.

[brinko99]

Thanks for the info...

Rather than building Uno myself; I played around with calling some of its internal methods directly as well as directly calling into libgl.so. This is what I'm using:

private static void OpenGlTesting()
{
    var assembly = Assembly.GetAssembly(typeof(GtkHost));
    var openGlRenderSurfaceType = assembly.GetType("Uno.UI.Runtime.Skia.OpenGLRenderSurface");
    var method = openGlRenderSurfaceType.GetMethod("TryValidateExtensions", BindingFlags.Static | BindingFlags.Public);
    method.Invoke(null, null);

    var property = openGlRenderSurfaceType.GetProperty("IsSupported", BindingFlags.Static | BindingFlags.Public);
    bool isSupported = (bool)property.GetValue(openGlRenderSurfaceType, null);

    var openGlEsRenderSurfaceType = assembly.GetType("Uno.UI.Runtime.Skia.OpenGLESRenderSurface");
    property = openGlEsRenderSurfaceType.GetProperty("IsSupported", BindingFlags.Static | BindingFlags.Public);

    // Uno calls this in OpenGLESRenderSurface.IsSupported
    // https://github.com/unoplatform/uno/blob/0bc77d0cbde8e7e4c79d3875f931a28623b769e4/src/Uno.UI.Runtime.Skia.Gtk/Rendering/OpenGLRenderSurface.cs#L29
    isSupported = (bool)property.GetValue(openGlEsRenderSurfaceType, null);

    var nativeContext = new Silk.NET.Core.Contexts.DefaultNativeContext("libGL.so.1");

    // Is libGL.so.1 found and accessible?
    // Uno calls this in OpenGLRenderSurface.IsSupported
    // https://github.com/unoplatform/uno/blob/0bc77d0cbde8e7e4c79d3875f931a28623b769e4/src/Uno.UI.Runtime.Skia.Gtk/Rendering/OpenGLRenderSurface.cs#L29C2-L29C2
    var isAvailable = nativeContext.TryGetProcAddress("glGetString", out var getString);

    // Uno calls this in OpenGLRenderSurface.TryValidateExtensions
    // https://github.com/unoplatform/uno/blob/0bc77d0cbde8e7e4c79d3875f931a28623b769e4/src/Uno.UI.Runtime.Skia.Gtk/Rendering/OpenGLRenderSurface.cs#L59C22-L59C43
    var gl = new GL(nativeContext);

    string version = new GL(nativeContext).GetStringS(GLEnum.Version);
    string extensions = new GL(nativeContext).GetStringS(GLEnum.Extensions);

    var hasARBVertexArrayObject = extensions?.Contains("GL_ARB_vertex_array_object");
}

For me, the calls to GetStringS return null.

[HakanL]

That's a good idea! What happens in my system is that GL is supported, GLES is not. However, none of the extensions are available (GetStringS is null). But what's odd is that everything I try to query from the GL instance comes back null/0, version, extensions, number of extensions. It seems that something is going on with Silk.NET. At one time I thought maybe it's a 32/64-bit issue, but I have tried this code on both now, and also on a standard Raspberry Pi 4. It comes back with only GL (without extensions) supported, but I can run various examples just fine. I'm currently testing this on Raspian 5 32-bit, so a standard install. FWIW the Silk.NET examples don't work either, but I believe it may be trying to use GLFW. But I can run GLFW examples (C++) just fine. To be continued...

[HakanL]

I created a separate issue to track native Wayland support in Uno here #14075. Gtk3-demo does it, so in theory it should be possible. I would be happy with GL support in Wayland, but GLES should also be an option, even though I don't know if it's better (faster) than GL. There's also GLFW that seems to be popular.