Offscreen rendering not using a control

[tcuthill]

Description

I am a Skia newbie so pardon if my question is based on misunderstanding or ignorance.

I work as part of a team that develops mobile apps and Xamarin and MvvmCross is the technology we use. The first project I worked on required GPU vector graphics to be run on Windows 8.1, so we couldn't use Skia and used Win2d instead. It turns out that MvvmCross will drop load events randomly. Given that the Win2d controls use the load and unload events to start up GPU connections, this caused the mapping widget which was shared in more than one view, to freeze and not be recoverable. The way around this was to use CanvasRenderTargets, which were not controls but had a GPU context held internally, and which is not affected by load and unload events. A service could then be built to generate maps based on vector features and pass the images to the control to render.

It seems that in SkiaSharp, the GRContext is hidden inside an internal library, and can only be accessed through a native view (eg. SKGLView). Ideally I'd like to be able to do offscreen GPU based rendering of vector based features, which would generate sources which can be rendered by controls. This question may be along similar lines to issue #665. Is this possible? It would be a great benefit to our project if it is.

Barring that, can I grab the glContext from the native view and will it be unaffected by actions on the view itself?

Any insights would be greatly appreciated.

Cheers,
Tom Cuthill

[mattleibow]

The simplest answer to all of your questions is yes :)

Both the native and forms views have a GRContext property that represents the drawing context on the GPU. You can use that, along with the SKSurface.Create overloads to construct arbitrary surface that you can draw on.

Once you have the surface, you can draw just like if it was from the view.

[tcuthill]

Hi Matt,

Just to clarify the approach then. I believe what you are saying is that the GRContext I can take from an SKGLView will survive the view being destroyed.

So the proposed image generator service will start up in its own thread, create a SKGLView to ensure that the GRContext is bound to the thread, grab its GRContext and then delete the SKGLView. Then as images are requested of it, it will do the following, as suggested in the doco for SKCanvas, constructing a GPU surface:

Then the required vector features can be rendered into the canvas, and the surface snapshotted to get the results as an SKImage. The SKImage can then be passed back to the view which requested it. That view should be able to render the image, even though it was possibly created in a different thread. (I'm a bit unclear about this part, is the SKImage a set of GPU instructions(I think it is?), or is it a bitmap. If it is instructions, then I have to convert it to a bitmap so that there are no issues going across threads?)

Also in Win2d there was a way of re-establishing the connection to the GPU device if the connection fails. Is that something that needs to be handled somehow?

Thanks again for you advice. I look forward to putting all of this into action!

Tom Cuthill

[tcuthill]

I've looked into some of the source code and it appears that when GL controls are disposed, the grcontext is also destroyed (eg. SKGLControl).

So if the map service is passed the grcontext instead from external controls, the controls themselves could be in the midst of being disposed of when the map service is trying to render using the grcontext. This would likely cause memory violations, right?

If the grcontext could be captured independently of controls then there would be no issue.

[tcuthill]

The approach doesn't work. Even if the SKGLView is created in the main thread, the GRContext is null:

[tcuthill]

Ok, I've looked into the source code and found that the GRContext is is fact, only created after the first request to draw on the surface. That means that the widget that will receive the images in the iOS world must be a SKGLView. So, it appears, that background rendering can not be separated from the control, which isn't ideal for an image generating service, but I'll do my best to get around this.

Taking this into consideration, I've held the GRContext associated with each SKGLView inside the service, and created a surface for each, which will service image requests, when they come in.

Now there appears to be a problem rendering with a surface from the GRContext. The first image generated is fine. Subsequent images snapshotted from the surface are always rendered as black.

I've been assuming that the SKSurface is reusable, and that SKSurface.Clear() would reset the surface back to the start so that it could get new drawing instructions. I thought that every time you get a canvas from the surface and draw into it, the surface accumulates the drawing. Am I misunderstanding?

Here is the code in the service. renderTarget.RenderingSurface is the SKSurface which has been generated from the GRContext.

The image callback is in the SKGLView. All it does is to catch the image and SetNeedsToDisplay():

Then in the SKGLView's OnPaintSurface, I just draw the image:

Is there something I should do to prepare the SKSurface to accept new drawing instructions (am I supposed to be flushing things out? If I call canvas.Flush() at the end of the DrawRect's it produces a segmentation violation. Also I've made sure to use SrcOver blending).

Any insights into this would be greatly appreciated. I'm up against a fairly tight deadline, and if I could just get offline rendering working, I'll be over the hump!

Cheers,
Tom Cuthill

[tcuthill]

Here is the class where I generate the surface:

[tcuthill]

Also, can I render into the generated SKSurface in a background thread? I looks like it is possible given what I saw in the UWP Swap-chain based widget.

[tcuthill]

I am suspecting that some kind of flush needs to happen. Sometimes only a partial image comes back. Does a Snapshot on the surface not guarantee that all the rendering is complete (ie. is it synchronous?)

[mattleibow]

I am looking at what you are trying to do and I think I may have got it: you want to create images at arbitrary times, and then render them on the screen at some point?

If this is the case, then you don't need the views at all since you can just create your context.

The thing to remember is that SkiaSharp does not initialize OpenGL. What you need to do is rather YOU initialize OpenGL on another thread, and then initialize SkiaSharp:

// make sure that OpenGL is initialized
// ...

// set up the SkiaSharp context
var grContext = GRContext.Create(GRBackend.OpenGL);

// create the GPU surface
var surface = SKSurface.Create(grContext, true, new SKImageInfo(100, 100));

// draw
var canvas = surface.Canvas;

The issue that you are facing is that the OpenGL context and the GRContext only live as long as the view, because they were created BY the view. But, there is nothing stopping you from just creating your own context at any time.

[pathw0rk3r]

We've been able to get the service working, the issue really was in trying to find out how to generate the GRContexts for each platform. Here is some help for anyone who needs to follow a similar approach.

For iOS it is relatively easy:

Add in the OpenGLES library in the 'using' section.

For UWP it is much more complicated. You need to use Interop services to get to the libEGL.dll.

Make sure to have "using System.Runtime.InteropServices" at the top of the class. Here is the class which makes use of the library to generate the context.

Hope this helps!

Cheers,
Tom Cuthill

[mattleibow]

Thanks for all this info @tcuthill. Some of the code (UWP) looks similar to what I had to do for the GL view - which is great. I was wanting to put some of this logic into a public API for SkiaSharp, but I am not sure if I want to maintain that just yet - at least iOS is easy 😄 Are you looking at Android at all? I wonder what you will have to do there?

[mattleibow]

I am closing this now as you appear to have things in the bag, but just wanted to ad another link to #755 because that may also be helpful. I need to looks some more, but that issue may be related to multiple threads, GRContexts or OpenGL contexts.

If you are reading this many years into the future and are having issues with multiple of something, then pop over there and you may have solutions 🤞