Enable Retained Rendering

[liyuqian]

This is for tracking all related pull requests for enabling retained rendering.

In my local experiments, the retained rendering can speedup fading transition by 2.15x and put our flutter_gallery__transition_perf average_frame_rasterizer_time_millis well below 16ms on Moto G4. Hence it will fix #13736

[liyuqian]

CC @Hixie @cbracken @chinmaygarde @jason-simmons @yjbanov @amirh

Let's try to close this issue before the end of October so we can squeeze another 30% speedup?

We'll also have to check the memory usage (thanks Chinmay for the remind). My guess is that the memory cost should be much less compared to our current raster cache: in my experiments, I only add the raster cache when there's an animated opacity. Those things should be much rarer than RepaintBoundaries and PictureLayers. Chinmay and I also think that we shall provide the raster cache option to the developers just like what we did with clipping.

[yjbanov]

we shall provide the raster cache option to the developers just like what we did with clipping

What would the API look like?

[liyuqian]

For example, the constructor of AnimatedOpacity could take one more parameter cacheBehavior which has 3 possible values Cache.always, Cache.never, Cache.auto.

[liyuqian]

Just found a bug in my experimentation which caused my program to never reuse the cache...

So the 30% speedup is just by replacing saveLayer with taking snapshot and then paint. So this part is memory-free.

After fixing the bug, the speedup is actually ~2.125x

[Hixie]

Can you elaborate on what "retained rendering" entails?

[liyuqian]

Yes, retained rendering in my current experimental implementation means

1. On the framework side, maintain which Layer is dirty, and which Layer subtree has dirty Layers.
2. On the engine side, make std::shared_ptr<flow::Layer> instead of std::unique_ptr<flow:Layer> and send such shared layer pointer to the Dart side during the SceneBuilder::pushXXX call.
3. During Layer::addToScene phase, check if the subtree is dirty or not. If so, proceed as before. If not, reuse the shared layer pointer (retained layer subtree) and immediately stop the subtree walk from there.
4. During the Paint/Preroll of some special engine layer (e.g., OpacityLayer), take a raster snapshot of the whole subtree, and reuse it if the whole subtree is retained and the matrix is unchanged other than translation.
5. For flow::PhysicalShapeLayer in Fuchsia, we can skip the Paint entirely if the subtree is retained and the matrix is unchanged other than translation.

[yjbanov]

Will changing AnimatedOpacity alone be effective or will we also require that the child of AnimatedOpacity be wrapped in a RepaintBoundary?

Also, we should probably be consistent with the methods we provide to developers for controlling the raster cache. For example, we already have a concept of willChange. Perhaps we should use the same flag in AnimatedOpacity. Alternatively, we could change the existing flag to use the Cache enum before we freeze the API. I'm leaning towards the enum, because it is extensible in the future. A bool flag will be hard to extend.

[liyuqian]

Child of AnimatedOpacity definitely won't have to be wrapped in a RepaintBoundary because we already automatically consider the Opacity as a RepaintBoundary.

I also prefer willChange to take an enum, and reuse it in retained rendering.

[yjbanov]

we already automatically consider the Opacity as a RepaintBoundary.

Do you mean the engine treats it as a repaint boundary or the framework? I'm not aware of anything in the framework that treats it as a repaint boundary, and so the framework will hand you a new Picture on every frame. Perhaps I'm not looking in the right place?

[liyuqian]

The framework has alwaysNeedsCompositing = true if Opacity has child and it's not fully opaque or transparent: https://github.com/flutter/flutter/blob/master/packages/flutter/lib/src/rendering/proxy_box.dart#L732

[yjbanov]

AFAICT, alwaysNeedsCompositing does not affect what the framework repaints. It only causes an opacity layer created for the Opacity widget. When the opacity changes the opacity setter is called, which in turns requests a repaint. This will repaint the entire subtree under the Opacity widget and generate a brand new Picture, unless there's a RepaintBoundary to stop it.

BTW, RepaintBoundary may not be sufficient. You need something that will prevent re-layouts from being propagated from the child to the Opacity widget. IOW, something that causes RenderObject._relayoutBoundary to be true, such as a SizedBox or something.

[liyuqian]

Oh, you're right. In my experiments, the RepaintBoundary is not provided by Opacity or AnimatedOpacity, but by the ModalRoute. It works for me because my AnimatedOpacity is always applied to a page transition between two ModalRoute.

Considering the cost of Opacity and AnimatedOpacity, I wonder if we should set isRepaintBoundary => true there just like other heavy RenderObject such as TextureBox and RenderView.

In any case, I did plan to add contraints to engine OpacityLayer so it should only have one child instead of multiple children. I thought that this was always satisfied because of the RepaintBoundary and its single child OffsetLayer. If this is not the case, then I'll have to make it so by introducing RepaintBoundary.

So to answer your original question, yes we'll have Opacity and AnimatedOpacity to be RepaintBoundary so their children is wrapped into a single OffsetLayer.

[Hixie]

I'm not sure I understand how the dart:ui API is expected to look after this change. Can you elaborate on this?

[liyuqian]

For CustomPaint widget, we'll have

class CustomPaint extends SingleChildRenderObjectWidget (
  ...
  this.cacheBehavior = Cache.auto
)

instead of

class CustomPaint extends SingleChildRenderObjectWidget (
  ...
  this.isComplex = false,
  this.willChange = false
)

as isComplex and willChange only affect raster cache behavior so it's probably more clear to just let developers decide whether to cache it or not.

Similarly, for Opacity/AnimatedOpacity, we could have

class Opacity extends SingleChildRenderObjectWidget (
  ...
  this.cacheBehavior = Cache.always (or, Cache.auto, Cache.never)
)

[liyuqian]

As discussed with @Hixie , we probably won't change the APIs of CustomPaint, and we'll probably add cacheBehavior to RepaintBoundary.

[liyuqian]

#23434 completes this feature request. We'll track more exploratory work in retained rendering in a new issue: #23535

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.