Decouple gles2 backend in rust-layers from android.

[glennw]

There are a lot of potential benefits to being able to use the rust-layers gles2 backend on desktop Linux.

In particular:

• Some desktop linux ARM SoC boards would be able to run Servo.
• Supporting Wayland/Weston (with some of the ongoing glutin work) may become a lot simpler.

[glennw]

Most of this work will involve changes to rust-layers (https://github.com/servo/rust-layers) which is a dependency of Servo, and https://github.com/servo/servo/tree/master/components/compositing (which is how Servo interacts with rust-layers).

Currently, we use #cfg() flags (these are somewhat similar to C preprocessor defines) to compile an OpenGL ES2 implementation of rust-layers for Android, and an OpenGL implementation for desktop Linux targets. (The Mac implementation uses the same technique to select an IOSurface implementation).

We would like to be able to use the ES2 implementation on both Android and Linux, and be able to select dynamically (via a command line flag) whether the desktop Linux version should use GL or ES2. This would allow us to run Servo on some ARM SoC boards, such as Raspberry Pi. It will also make it easier to support Wayland in the future.

Prerequisites:

• You need to be working on a Linux desktop that supports GL ES2 - any modern Linux desktop with a recent Mesa should be fine for this. However, it may not be possible to do this work from a VM. Running glxinfo should report "OpenGL ES profile version string: OpenGL ES 2.0 Mesa 10.6.3" or similar.

The basic steps involved are:

• Add an option to opts.rs to allow selecting whether to use GL or ES2 for the rendering backend.
  • This will only have an effect on desktop Linux builds.
• Pass this option from the compositor through to rust-layers during initialization.
• Change the platform implementation in https://github.com/servo/rust-layers/blob/master/src/platform/surface.rs
  • Keep the #cfg() flags for Mac (since that can always be selected at compile time) but remove the Linux/Android specific #cfg flags.
  • Instead, provide a generic GL implementation that can dynamically select between GL or ES2. (this could be placed on a #cfg() flag for Linux and Android targets)
  • There is likely to be some changes required to the EGL platform/surface.rs implementation to make it work on desktop GL, depending on extensions present.

[mrobinson]

I'm really excited about this and am willing to help out with reviews and refactoring.

[mrobinson]

This might also be useful for this work: https://github.com/waffle-gl/waffle

[R00ney]

Hello, this is Neal from the GLE_ES2 refactor team. We have some clarification questions we would like to ask.

Whether waffle-gl is to be imported, or simply used as example for base functionality?

Where platform/android/surface.rs implements ES2 vs GL (ie, what is the dividing line between a GL graphics implementation, and an ES2 graphics implementation).?

Would be possible to acquire a video of a demo of the Android GL vs ES interface? We have not yet seen the android servo in action.

[mrobinson]

@R00ney, I think that waffle-gl could be used either as a dependency or as inspiration for how to do this in Rust. I'm not very familiar with it, only knowing that it does something similar.

For C/C++, the difference between OpenGL and OpenGL ES is in the particular library and header that you use. This also varies based upon the particular platform you are on. The situation is a little complicated. :)

Currently, Android is using OpenGL ES, so I don't totally understand your question about a video demo of Android using the desktop OpenGL interface. Compiling Servo for Android is not overly complicated (but it helps if you've used the Android SDK before). You can find some decent instructions here: https://github.com/servo/servo/wiki/Building-for-Android

Let me know if you run into any issues. I'm mrobinson in the #servo IRC channel.

[glennw]

I haven't looked at waffle in depth, but I think it's probably better to look at that as a possible reference, rather than use it as a dependency.

Right now, the android implementation (android/surface.rs) is ES2 only, and the Linux implementation is GL only, so I guess that is the dividing line.

Feel free to ping me on irc (gw) or reply here for any further questions!

[R00ney]

Hello,
will either of you gentlemen have time to be on irc (either servo or gw) this weekend?

I believe we are still having some issues understanding how the current code calls/includes GL or GL_ES2, and thus are struggling with clarity of this refactor task.
For example, a code compare of the two surface.rs files for the android and linux folders quickly show a large number of implementation difference, making refactoring them with a simple Graphics setting choice feel like a tricky task.

Obviously any advice is appreciated, and please let us know when you are available to chat.
Thank You

[glennw]

Hi, I'll be available for some portion of the weekend (although probably in a different time zone to you). My IRC client is always connected though, so any messages you send to me there will get answered eventually :)

[mrobinson]

@R00ney My IRC client is similar to @glennw's so I can also answer question asynchronously, which I'd be more than happy to do.

[R00ney]

Maybe I'm using IRC wrong then, I go to mibbit (https://client01.chat.mibbit.com/) and select channel #servo or #gw and so far haven't seen anyone. Am I using the wrong settings? Thanks you.

[glennw]

My irc nick is gw, we're in the #servo channel - did you join the mozilla irc server (you need to select this from the server dropdown on the mibbit page)?

[jdm]

Note that if you're using a client that is not permanently connected to IRC like @glennw's, you can see logs of public messages in the channel at http://logs.glob.uno/?c=mozilla%23servo .

[prashantgupta24]

Hello all, we have a couple of questions concerning the code changes we wish to make.

We have a local flag in render_gl.rs (https://github.com/prashantgupta24/rust-layers/blob/GLES2/src/rendergl.rs, line 499) that is our command line enum variable selecting the GL or GL_ES2 graphics options.
What is the best method for making this variable accessible to surface.rs ?
Some sort of global variable? If so, can you comment on how to make global variables in this rust, previous attempts we made to make it global did not work.

Secondly, if we understand correctly, in surface.rs we wish to replace the
#[cfg(target_os="linux")] and #[cfg(target_os="android")]
lines with our graphics_select flag?
Or is our graphics_select flag supposed to be within the cfg flags, to make the function calls be GL or ES2?

We are having difficulty understand how this would be feasible, given that the cfg flags run a compile time, while our flag would have to run at runtime, and not all of the differences from the android and linux seem to be mergable, or related to the difference between GL and ES2.

Additionally, as we understand it, GL_ES2 is an emebbed system graphic library, yet our flag is to potentially select this graphics option even if we are on a linux system. Is that functionality valid?

Any resources you could point us towards to understaning the merge of linux GL and android GL_ES2 would also be helpful.

Link to my Servo repo: https://github.com/prashantgupta24/servo

Link to my Rust-layers repo: https://github.com/prashantgupta24/rust-layers/tree/GLES2

[glennw]

Hi,

I think it might be best to add a parameter to the NativeSurface constructor, which specifies which graphics API to use (GL or ES2). This could then be passed from the servo calling code in create_layer_buffer, based on the selected graphics API from the command line. @mrobinson knows this specific code much better - so let's check with him if that makes sense, or he would suggest a different method.

You are correct in thinking the graphics select flag won't work as a compile time flag - we definitely want to be able to select whether to use GL or ES2 backend at runtime. Effectively, at the end of this task - I imagine we'll have #[cfg(target_os="linux")] on the GLX/X11 specific code implementation, and #[cfg(any(target_os = "linux", target_os = "android"))] on the ES2 specific code implementation, since we want it available on both Android + desktop Linux.

You're right that ESv2 is primarily an API for embedded systems, but it's also available on (most) desktop Linux installations now. This is particularly useful for a couple of reasons, for example, allowing us to run the ESv2 backend on Linux makes it a lot easier to debug ESv2 problems rather than debugging directly on Android. More importantly, there are some desktop Linux platforms (such as Raspberry Pi) that run desktop Linux, but only support hardware acceleration when you are using ESv2.

Perhaps don't worry too much about the runtime selection for now - a good proof of concept would be to comment out the existing GLX specific code in rust-layers, enable the current android/es2 code in rust-layers - and get that compiling and running on desktop Linux. That would ensure that the most important goal is achieved (that we can run Servo with ES2 on desktop Linux), and then after that is working, we could discuss the cleanest way to integrate it to allow runtime selection.

Feel free to ping me on IRC (gw) or reply here if that doesn't make sense or you have follow up questions :)

[jdm]

Given that the only use of NativeSurface's constructor is WorkerThread::create_layer_buffer in components/gfx/paint_task.rs, that makes the command-line flag stuff more straightforward than I originally anticipated. Specifically, I think my original suggestion to add an option to RenderContext::new is unnecessary, since there doesn't appear to be code in rust-layers that invokes the NativeSurface constructor. This suggests that we can keep all of the code that deals with the command-line flag inside the code in Servo.

[djain2405]

So following up on your previous suggestions, we are trying to merge the code from the android platform surface.rs into the linux platform surface.rs. But we are facing a lot of conflict issues when we are trying to merge the code from the android to the linux one. So we have a couple of questions regarding that:

1. Do we need to copy the implementation of the EGLImageNativeSurface from the android surface.rs into the linux surface.rs , and if that is the case, how do you suggest we merge the NativeDisplay implementation? While attempting to merge the android surface.rs to linux surface.rs, we've esentially compied android into linux file and renamed the NativeDisplay to NativveDisplayEgl, just to avoid duplicate name conflicts. Then modified the src/libs.rs and the Cargo.toml to allow egl imports for linux systems to fix several imports.
   However now, the method egl_image_target_texture2d_oes from gleam is behind an cfg(target_os=android) flag.
   We suspect this will hold true for more than just this fucntion, so we'd like to know if it is neccesary to clone gleam, change the cfg, and make servo use the local gleam, or if there is a better global refactor method to allow the linux version of servo to use more android methods? When we tried to merge the code If not, then do we need to rename the NativeDisplay method for android and then copy it to the linux surface.rs to ensure EGL is supported for linux too.
2. Is there any way in rust to use an existing file instead of copying the code. What we mean to ask is is there a way we can use the android surface.rs for linux too by referencing the file instead of copying the code from that into the linux surface.rs?

Any suggestions or tips would be extremely helpful.

[R00ney]

@glennw , would it possible to chat with you on #servo (or anywhere,really) around 1pm EST on Sunday about the issue posted ?

[glennw]

@R00ney Sorry, that was 4am for me and I missed it. It looks like you got some answers to your questions on irc - but I'll try to be around on irc a bit for the next few hours if you have more questions.

[R00ney]

@glennw For the moment, we just have the questions listed above. However, when we work on it tomorrow, I'm sure we'll have more questions. Do you have a best time you're available Sunday afternoon?

[jdm]

For context for everyone - @glennw is in UTC +10; @R00ney is in UTC -5.

[glennw]

@R00ney

https://www.timeanddate.com/worldclock/meetingtime.html?year=2015&month=11&day=29&p1=377&p2=47

I'm generally available from around 5am on Monday, which would be 2pm on your Sunday - so we could try to chat then if that works for you.

I'll be in and out today, but I'll get some answers to your questions written up before then. @mrobinson is also very interested in this work and is in a somewhat better timezone - so it may be worth seeing if you can ping him on irc if I'm not there.

[glennw]

In terms of the egl_image_target_texture2d_oes function - I had a quick look in gleam and that's the only function that is linked on android, but not Linux.

So a good first step would be to change the definition of that function in gleam from:

#[cfg(target_os="android")]

to

#[cfg(any(target_os="android", target_os="linux"))]

If you do that, bump the version number in the Cargo.toml for gleam and open a PR, we can get that merged - then that function will be available on Linux.

In terms of re-using the android source code, and how to merge them - what I'd suggest for now is just copying the entirety of the android/surface.rs implementation into the linux/surface.rs file (replacing what's there now).

If we then work through solving the compile errors and issues that result from that, we'll have a solution that allows using EGL on Linux (but not GLX). This is useful as an intermediate step to ensure that we can get the EGL rendering working on Linux.

Once that is working correctly, we could discuss how best to structure the code to allow both GLX and EGL to exist at runtime on Linux (at that point, it's mostly just going to be moving some code around and renaming things - the tricky bits will be done).

I've just gone through that process here, and it's now compiling locally - although I've stubbed most functions out with panic!("TODO"); for now just to get it compiling (so it doesn't do anything actually useful yet).

Let's try and find a time to chat on IRC based on the times above to discuss in more detail - but getting the gleam changes done first and a PR for that merged would be a great first step.

[jdm]

I think as written this is solved now.