Strategies for static content rendering performance

[matthew-carroll]

I'm rendering document pages, which seems to be very performance intensive. I'm not entirely sure why. The volume of content in the document pages isn't dramatic. Nonetheless, the time to push the display list to the engine takes many frames.

Putting aside the time to push the initial display list, is there a way to cache a document page's display list in the engine, such that the page can be scaled up/down, without sending over another display list? The page content doesn't change after the initial render, only the ancestor transformations, so I would hope that there's a way to avoid any additional RenderObject work, but I haven't found such a path.

I tried wrapping each page with a RepaintBoundary, but this seems to yield very strange behavior when scaling up/down. Does the display list within a RepaintBoundary apply ancestor transformations up front, and then retain global coordinates? Or does the display list within a RepaintBoundary retain local coordinates and then play those on top of ancestor transformations in the engine? If the former, that might explain the weird behavior that I'm seeing.

Here are a couple videos that show the RepaintBoundary problem. In this first video, I have a series of "pages" that I can scroll and scale. The pages changes size as expected, they continue to meet exactly where they should, and the circle in the center grows and shrinks as expected.

pages_without-repaint-boundary.mp4

In this next video, I took the exact same code and added a RepaintBoundary around a CustomPaint, which paints the circle at the center of each page. Here's what happens now.

pages_with-repaint-boundary.mp4

These pages are laid out in a custom RenderObject, so it's possible that I'm screwing something up, but in the absence of a RepaintBoundary, the children appear with the offset and scale that they should.

So that's a problem that's actively preventing me from using a RepaintBoundary, but I'm also wondering if there are any other caching strategies that would allow content transformation without regenerating the display list for each page?

[matthew-carroll]

@chinmaygarde @jason-simmons - You probably know the answer to this.

[zanderso]

Hi @matthew-carroll, I'm curious about how the display list diagnosis was reached. If there was already a discussion going on another github issue or discord thread, would you mind linking that here?

cc @flar

[flar]

@matthew-carroll we need a code example to diagnose this.

[dnfield]

Specifically on the code example, it'd be really good to have something that shows the bad behavior with the repaint boundary you're seeing. If that's a bug in the framework or engine we'd like to fix it.

On the broader question about performance it'd help to see some tracing and understand what's slow. It's hard to answer in the abstract. Maybe a strategy that preserves picture layers (like what flutter_svg currently does) would help. Maybe you need an API that lets you render text without using the whole of the paragraph machinery on each frame. Maybe you need to do something like use Picture.toImageSync to handle your own raster caching. Or maybe it's something else entirely. Without traces it's hard to say.

[matthew-carroll]

I'm curious about how the display list diagnosis was reached

Here's a thread dealing with the long-running display list transfer:
#114306

That ticket focuses on the fact that it takes a long time to push the display list. It would obviously be a big help to find a way to eliminate the time for all display list pushes. This ticket is looking more at mitigation strategies to eliminate all unnecessary recreations of that display list.

It's hard to answer in the abstract.

I'll need to figure out how to share useful snippets of code. I'll chat with my client and see what we can do.

[flar]

I'll need to figure out how to share useful snippets of code. I'll chat with my client and see what we can do.

I'm not sure we need the entire original application, but the code sample that were used to make those videos - the ones that showed the transform bugs - would help us diagnose the problem you are getting when you add the RepaintBoundaries.

[dnfield]

Preserving the display list won't help you here, since what's actually expensive is rasterizing all the text in the displaylist.

It sounds like what you're looking for is Picture.toImageSync, although with text that can get a bit strange if you have to recomposite it with alpha against some other background.

It would still be nice to know more about the bug in the video around the repaint boundaries.

[matthew-carroll]

It would still be nice to know more about the bug in the video around the repaint boundaries.

I think I've confirmed that the layout issue was my fault. I was doing something that wasn't quite right in terms of canvas vs layer transforms. I think I've fixed it.

I still plan to bring more material to this ticket, as requested. I need to reach an appropriate stopping point with my current work so that I can figure out what to extract and post here.

[flar]

I tried wrapping each page with a RepaintBoundary, but this seems to yield very strange behavior when scaling up/down. Does the display list within a RepaintBoundary apply ancestor transformations up front, and then retain global coordinates? Or does the display list within a RepaintBoundary retain local coordinates and then play those on top of ancestor transformations in the engine? If the former, that might explain the weird behavior that I'm seeing.

I just saw this question within the original description. When recording to a Canvas, say in a CustomPaint or any RenderObject that executes calls on a ui.Canvas object, it is recording local coordinates. There is no initial transform in the ui.Recorder or ui.Canvas. The transforms in parent Widgets/RenderObjects are expressed by enclosing Transform objects and the recordings within a given Canvas are played back relative to those transforms.

If you have a RepaintBoundary around the generation of that display list, then the contents of that RepaintBoundary are severed from any connection with the surrounding painting and that display list is not regenerated unless a Widget underneath the RepaintBoundary widget indicates some need to repaint (such as an animation). It couldn't contain global coordinates otherwise it couldn't do its job of avoiding the rebuilding of the display list (or in some cases display lists) that its children generate.

[matthew-carroll]

Update: We've done a lot of optimizing for document rendering.

Layout
There are so many operations to process per page of a document that we can't fit the layout process into a single frame. As a half measure, we run as many layout operations during the layout phase as possible, until we reach 8ms of work, and then we punt to the next frame. This approach takes a lot of extra frames for setup and teardown, and the time estimations aren't good enough to completely eliminate jank, but it's far superior to our previous implementation that blocked the main thread for the entire layout pass.

When we run layout, we're caching Path objects. We'd like to cache the layout on a background thread, but the Flutter engine won't let us create Paths or Paragraphs in a background isolate. It would be nice if these weren't tied to the main isolate. What we're going to try, instead, is to define our own data structures for various paths, so that we can collect those path definitions in a background isolate, pass them to the main isolate, and then convert them into Flutter Paths. This re-definition of paths and the serialization is a consequence of Flutter limiting where those objects can be instantiated.

Painting
We discovered that our primary performance impact with painting was the shear number of Paragraphs we were creating and dispatching, because the document format tells us where to place smalls runs of glyphs, rather than providing a large piece of text to flow, ourselves.

The best answer here would be for Flutter to expose some of the lower level text painting abilities in Skia. We'd like to be able to say something like canvas.drawGlyph().

Currently, we've worked around the Paragraph problem by integrating a font package that renders individual glyphs. We've used that package to paint packed glyph textures. And we're drawing glyphs to the screen using canvas.drawAtlas() for an entire page at a time.

Rasterization
One area that seems to be completely outside our control is rasterization. For some reason, we're seeing a rasterization time of 25ms to 40ms. It's unclear why this is happening. I don't think our total number of paths is much larger than other complex Flutter UIs. I thought that perhaps the raster time was due to drawAtlas(), so I removed drawAtlas() and the raster times were still far too high. We've also noticed that anywhere from 33% to 50% of the raster time is spent on the Android buffer swap command. For example, we have seen buffer swaps that report a time of 20ms.

Shader Compilation
We were seeing massive shader compilation jank on the first frame per page. I opted to precompile those shaders. As a result, most shader compilation time is gone, but for some reason there are still a number of small shader compilation steps for the same pages in the same document. Also, for some reason, the time that we saved from shader compilation seems to have been replaced with other things, as mentioned above in the "Rasterization" section.

Summary
We'd love to see some kind of mechanism to paint our own glyphs. We're reinventing a lot of stuff to get around Paragraph.

We'd love to see the ability to creating Canvas painting objects like Paths and Paragraphs in a background isolate.

We don't know what to do about the raster process. We can't explain why it's taking so long and we're not sure what we can do about it.

[zanderso]

Some related issues:

• Severe performance issues when drawing transformed widgets that contain beziers. #78543 <- The discussion in here may provide hints for investigating the raster time, but a pared down repro would really be the most helpful thing. Maybe the complex paths are hitting a software path in Skia?
• Ability to use ParagraphBuilder & Paragraph in other isolates #53985
• Enable dart:ui APIs for text measurement in secondary isolates #30604
• Using dart:ui from non-main-thread Isolates crashes #10647

[knopp]

Out of curiosity, how large is the text layout? Does it cover visible area, or entire document?

[matthew-carroll]

Out of curiosity, how large is the text layout? Does it cover visible area, or entire document?

At the moment, we're rendering an entire page at a time, regardless of visible area. That said, such a thing needs to be possible, anyway, because if the user zooms out, then the user is going to see the entire page. So aggressive culling might be a nice late-stage optimization, but lagging when the full page is visible isn't really acceptable.

[knopp]

I might be a bit confused with terminology. What exactly is a page? Text layouts usually works with documents and paragraphs. My question is, whether the entire document is laid out, or just the paragraphs currently being visible.

The problem I see here is that no matter how well you optimise the layout, if you keep laying out the entire document you are bound to reach the size where you just can't do that at interactive speeds.

For example TextKit 1 used to layout entire document by default and partial layout was an opt-in. In TextKit 2 partial layout is the default and laying-out anything outside visible viewport need to be manually requested. Considering partial layout a late-state optimisation seems a bit strange to me.

[matthew-carroll]

To be clear about terminology:

• page: a finite visual area filled with text and shapes, e.g., an 8.5"x11" area scaled by the DPI
• document: a series of pages

While rendering multiple pages in a document at the same time will certainly exacerbate performance issues, this ticket refers to issues that are faced when rendering even one page at at time. My point above is that we shouldn't have any issue rendering a single page, at a typical page size. We can open any popular document viewing or editing application and we can view a single page without issue. That page isn't culling any content, because all of its content is visible.