Custom draw operations, exposed SkCanvas and reduced Skia api visibility

[kekekeks]

Added a way to consume SkCanvas directly

[kekekeks]

#2364

@MarchingCube

[Gillibald]

Just an idea to avoid implementing a custom operation every time I want to have direct access to the context.
Render is called two times when the context is a DeferredDrawingContext. The first call add a special node to the scene graph that makes sure Render of the Control is called a second time when the actual drawing context is used. For ImmediateRenderer this is only called once.

public override void Render(DrawingContext context)
{
    if(context.IsDeferred)
	{
		context.RegisterDirect(this);		
		return;
	}
	
	var skiaContext = context as ISkiaDrawingContext;
	
	var canvas = skiaContext.Canvas;
}

[kekekeks]

How exactly would bounds checking and hit testing work with that API?

[Gillibald]

Where in the code base does a render call change bounds or the hit testing? I should probably have a deeper look at the rendering process. You can always override the controls hit testing behavior by implementing ICustomSimpleHitTest.

[kekekeks]

Draw operations are responsible of visual hit testing. Draw operation bounds are needed to detect the dirty area. Please, read DeferredRenderer's code base.

[Gillibald]

So ImmediateRenderer is currently not able to hit test because it doesn't use draw operations? Why isn't a visual responsible for hit testing against its own visual representation? Maybe that's why ICustomHitTest was introduced. I will not bother you more on that topic. Custom visuals with arbitrary shapes that need proper hit testing will need to implement a drawing operation. In my opinion a visual should know its visual representation and how to hit test against that and not some external wrapper.

[kekekeks]

Yes, immediate renderer has issues with hit testing. Due to the lack of the scene graph its hit testing is inaccurate.
Also visual's visual representation doesn't have to occupy the entire bounds of the visual, so for proper invalidation and dirty rect tracking we need the bounds of individual drawing operations.

[grokys]

Is this an alternative to #2364 or are they to be reviewed together?

[grokys]

Assuming this is an alternative to #2364, I think I actually prefer #2364: this adds core APIs that will need to be maintained ~= forever for very a niche functionality. #2364 on the other hand just modifies Avalonia.Skia which, being a platform implementation we can say that the API isn't guaranteed to be stable.

[MarchingCube]

@grokys Actually #2364 tackles different problem - being able to provide custom Skia GrContext to the Avalonia Skia platform.

This PR addresses the issue that user can't access raw Skia Canvas to implement platform specific drawing from the visual. With this we can utilise full Skia api, and render completely custom elements.

[grokys]

Ok, thanks for the background @MarchingCube. I still have a few reservations about the API here: adding a core API which then requires casting to a platform-specific API to do the drawing seems a little clunky.

As an alternative API (and not thinking about implementation details yet): what if in Avalonia.Skia there was a SkiaCanvas control which had a void Render(SkCanvas canvas) method, or even a Render event to which an SkCanvas was passed.

This render method/event would be fired on the render thread, so there would probably need to be an Update method which could be used to copy styled property values to local values.

[Gillibald]

I think we should do the same that WPF does and have a special Bitmap implementation that supports this. This has some overhead but is a lot cleaner.

[kekekeks]

@grokys
It's meant for extending the rendering context. With a custom drawing operation people can write extension methods for our DrawingContext. I think we should expose Direct2D render target in the same way.

@Gillibald
Let's say I have 1000 semi-transparent controls which render a custom line overlayed on top of each other on a full screen window. Your suggesion with a custom bitmap per control would consume 1920*1080*4*1000=8294400000 (8GB) of RAM. A custom drawing operation would consume almost no RAM

[kekekeks]

@grokys You can also think about custom drawing operations as means to explicitly access the immediate drawing context even when using a deferred renderer. This can be really useful if application has tons of stuff on the screen and doesn't care about the hit testing (i. e. a filled rectangle with thousands of lines). The scene graph for these non-hit-testable elements would just consume resources.

[Gillibald]

I understand the benefits of this approach I just don't like the DrawOperation itself. Can't we just introduce a "RenderDirect" method that is always called with the actual drawing context?

[kekekeks]

Not every drawing operation that is used by visual needs direct access to the immediate context. Some operations could be our built-in ones, some could be custom ones. With custom drawing operations one can mix and match those, with extending Visual's API one can't. I think you are viewing the API from a way too visual-centric point of view. DrawingContext could be potentially used even without the visual tree.

[Gillibald]

I agree that DrawingContext is Visual independent but hit testing is not. And this approach was used to support custom hit testing. I just think we should fix hit testing for all renderers and also make this kind of customization possible. This is just my opinion.

[grokys]

After thinking about this a little more, I think this is pretty much needed at a low level for the deferred renderer. It should probably be wrapped in something a bit more high-level that deals with the immediate renderer etc though moving forward.

Only thing I'm not sure about is DrawingContent.DrawCustom: longer term I think we should allow tapping straight into the scene graph using in a manner similar to UWP's composition API which would allow us to handle invalidation properly too.

[Gillibald]

For WPF they store the content(IDrawingContent) within the UIElement(Visual) after measure and arrange.

That content can be anything that can be processed by the renderer.

For Avalonia IDrawingContent could be a graph of DrawingOperations. Each Visual is responsible for hit test bounds etc.

The renderer requests the drawing content of a Visual when it's time to render something.

Each UIElement creates its own DrawingContext and produces IDrawingContent when the DrawingContext has finished.

[grokys]

@Gillibald I don't think that's true, or at least that's not the lowest level. From https://docs.microsoft.com/en-us/dotnet/framework/wpf/advanced/wpf-architecture:

WPF displays data by traversing the unmanaged data structures managed by the milcore. These structures, called composition nodes, represent a hierarchical display tree with rendering instructions at each node. This tree, illustrated on the right hand side of the figure below, is only accessible through a messaging protocol.

When programming WPF, you create Visual elements, and derived types, which internally communicate to the composition tree through this messaging protocol. Each Visual in WPF may create one, none, or several composition nodes.

This composition tree is what we call the scene graph, and the equivalent in UWP is the composition API I mentioned above.

[Gillibald]

I came to this by reading the source code. The only part that is not available is the native part of the rendering stack. Maybe I am wrong about the requesting part and the content is send to the renderer pipeline the moment it is created. In general WPF content that is being rendered is represented by a subclass of Drawing. In our case this is a DrawingOperarion. Clearly there is more to it on the native part. Having the content of a Visual as one unit would make it easier to apply effects on it. As I understand currently each drawing operation is treated as a unit.