Severe performance issues when drawing transformed widgets that contain beziers.

[modulovalue]

The video below shows a CustomPainter that draws hundreds of cubic beziers.
The CustomPaint Widget is transformed using an InteractiveViewer Widget. After a certain zoom level the performance degrades severely from 4ms to 80ms+

Bildschirmaufnahme.2021-03-18.um.15.32.03.mov

import 'package:flutter/material.dart';

void main() => runApp(MaterialApp(
      home: InteractiveViewer(
        child: CustomPaint(
          painter: MyPaint(),
        ),
      ),
    ));

class MyPaint extends CustomPainter {
  const MyPaint();

  @override
  void paint(Canvas canvas, Size size) {
    final path = Path();
    for (int i = 0; i < size.width - 50; i++) {
      final dx = i.toDouble() + 20;
      path
        ..moveTo(dx, size.height * 0.05)
        ..cubicTo(dx, size.height * 0.3, dx + 5, size.height * 0.5, dx + 10, size.height * 0.95);
    }
    canvas.drawPath(
      path,
      Paint()
        ..color = Colors.blue
        ..strokeWidth = 0.3
        ..style = PaintingStyle.stroke,
    );
  }

  @override
  bool shouldRepaint(covariant MyPaint oldDelegate) => false;
}

[✓] Flutter (Channel master, 1.26.0-18.0.pre.2, on macOS 11.2.2 20D80 darwin-x64, locale de-DE)
    • Flutter version 1.26.0-18.0.pre.2 at /Users/valauskasmodestas/Desktop/flutter
    • Framework revision fa670e0b5e (10 days ago), 2021-03-08 15:25:40 +0100
    • Engine revision 2c527d6c7e
    • Dart version 2.12.0 (build 2.12.0-259.8.beta)

[pedromassangocode]

This is somewhat related to #72718.

[chinmaygarde]

cc @flar

[flar]

This happens at the transition from hairline stroke widths to non-hairline stroke widths. At rest the stroke width is 0.3 pixels which is less than a pixel and a fast algorithm that draws paths that stroke only single pixels is used. As soon as the scaling causes that stroke width to exceed a pixel in size, complex geometric calculations must be performed to determine which pixels are affected by the stroking and by how much. Non-hairline stroking is a complex operation that comes at a large cost.

Changing the strokeWidth in the example to 0.0 eliminates the performance loss, but the results will look different. An alternative rendering to try would be to use a fill operation between adjacent cubics that are separated by the desired "stroke width" from each other. It will approximate a stroke without all of the expensive calculations. If the cubics are mildly curved enough, it will be nearly indistinguishable from a stroked version. (Filling between pairs of cubics is actually worse - it looks like much of the difference is just whether Skia can use its really fast hairline renderer or a path renderer.)

Here are some screen shots of the tracing showing the difference in the operations:

Fast cubics using a hairline renderer in Skia

Slow cubics using a path renderer in Skia

[ibrierley]

@flar thanks for the feedback on the issue. I'm looking into similar at the flutter_Map vector support issue Git referenced previous to this, and I think we do get the same problem. Thin strokes are really quick, fatter strokes really slow.

I'm just trying to figure any workarounds for this. Do you know if the problem is simply that the stroke is thick ? Or is it that it is transformed ? I.e I've been playing with pretransforming the paths (rather than for example a transform on the canvas), but it didn't seem to make any difference.

Do you have any thoughts in design or workarounds that would avoid even hitting the problem (other than having hairline stroke widths only) ? Just wondering what other none flutter canvas solutions do to get around the problem ?

[flar]

@ibrierley the hairline renderer does something really fast that can't really be "pre"-pared out of it and it can only be used when the width of the path is less than a device pixel. It's basically the difference between grabbing a crayon and dragging it from point A to point B vs. drafting the edges of the "stroke" and then using the same crayon to carefully fill inside those edges. One action is nearly instantaneous, the other involves elbow grease and sweat.

Whether a stroke is a hairline or not is basically "transform the width and see if it is >= 1.0 pixels". So, it's a product of stroke width and the scale of the transform.

About the only way to keep in the "fast path" would be to try a stroke of 0, but when you zoom that in really far you will see the difference since the hairlines won't fill the space as much as a non-hairline would.

[ibrierley]

Thanks a lot for the explanation. I think it's useful to know as there may be a few small optimisations I can make using it.

I'm guessing there's no way to force it off (at the expense of accuracy), I had wondered if during animations and zooms it could have been forced off somehow (as you care less then), and then when settled switched for accuracy if that makes sense. I suppose I was thinking about things like SVGs "crispEdges" vs "optimizeSpeed" that has that trade off.

[flar]

Just to be clear, "forcing it off" is setting the width to 0. That seems to make it fast regardless of scale.

[tmaihoff]

I had the same issue with my CustomPainter. I need to draw hundrets of coordinates on the screen. With a Path(), this resulted in horrible lags when I wrapped it into an InteractiveViewer and tried to zoom or pan (worse lags when zooming).

The solution for me was to switch from path drawing to line drawing.

This might not work for every use case and you can't fill the drawn area any more, but for me that worked and eliminated the lags.

Just for the sake of completeness, this is what I did before which caused lags:

for (int i = 0; i < pointList.length; i++) {
  if (i == 0) {
    path.moveTo(pointList.first.dx, pointList.first.dy);
  } else {
    path.lineTo(pointList[i].dx, pointList[i].dy);
    if (i == pointList.length - 1 && closeShapes) {
      path.lineTo(pointList.first.dx, pointList.first.dy);
    }
  }
  canvas.drawPath(path, linePaint);
}

This is what I do now -> no more lags

for (int i = 0; i < pointList.length; i++) {
  if (i != 0) {
    canvas.drawLine(pointList[i - 1], pointList[i], linePaint);
    if (i == pointList.length - 1 && closeShapes) {
      canvas.drawLine(pointList.last, pointList.first, linePaint);
    }
  }
}

[flar]

I had the same issue with my CustomPainter. I need to draw hundrets of coordinates on the screen. With a Path(), this resulted in horrible lags when I wrapped it into an InteractiveViewer and tried to zoom or pan (worse lags when zooming).

The solution for me was to switch from path drawing to line drawing.

This might not work for every use case and you can't fill the drawn area any more, but for me that worked and eliminated the lags.

Drawing a path with a lot of lines is time consuming as the stroke geometry has to be calculated. Did you try drawPoints as well?

Here are the things you give up when drawing a bunch of line segments instead of making a path (or drawPoints polygon):

• You can't fill the shape as you said
• The "join" geometry at the corners is lost, but you can make it less of an issue with round caps that look similar to round joins on a polygon
• You get double rendering at the corners which can have 2 effects
  • [comment updated]: also if the lines cross each other you get overdraw at the crossings with all of the issues below
  • lower quality antialiasing as the feathering at the edges is overdrawn (it's like bad antialiasing where the coverage of those edges is miscalculated as double the coverage it should have)
  • reducing opacity of the color of the lines makes the corners less transparent than the rest of the lines unless you use a saveLayer to implement group opacity which costs a lot as well

[ibrierley]

This is interesting, I had been looking for a drawLines method but couldn't find one (and was going to request if it could be exposed), had assumed drawLine would be slower due to needing multiple instructions, but maybe not. I'll probably give this a test when I get a chance as I draw a lot of lines.

I'm a little confused though...why does a drawPath have stroke geomoetry to be calculated but multiple drawLine doesn't (if that is the case and assuming here we're talking about multiple lines and not curves), is this just because of corner joins ?

Is there a possible optimisation to be done by exposing a drawLines option (i.e one instruction, lots of points, less geometry to be calculated, if you don't need fill or any of the other parts flar suggests)?

[tmaihoff]

@flar You're right about the drawbacks, but at least for my specific usecase the performance improvement is the clear winner. Now, I only have to sacrifice some styling options, before (with path) scaling was not possible at all.

And yes, I use canvas.drawPoints as well, which can be scaled as well without problems.

@ibrierley Yes, the CustomPainter's canvas has the method drawLine

[flar]

I'm a little confused though...why does a drawPath have stroke geomoetry to be calculated but multiple drawLine doesn't (if that is the case and assuming here we're talking about multiple lines and not curves), is this just because of corner joins ?

The geometry of a single drawLine can be easily hardcoded. If the endcaps are BUTT or SQUARE then it is a simple rectangle aligned with the direction of the line. If the endcaps are ROUND, then it is a rounded rectangle aligned along the direction of the line. A few instructions just dispatches the appropriate primitive rendering.

The geometry of 2 lines in the same path invokes a general algorithm which follows the path, computes joins and caps which involves a bit of trigonometry at each vertex for the joins, etc. It also expresses the outline of all the geometry in the path in a single resulting path so that it is filled in one single operation rather than in a number of separate operations so that there is no overdraw of any of the pixels resulting in the issues described above.

Even if the path could identify "hey, I'm actually just a list of moveTo/lineTo pairs", that wouldn't be enough to alleviate the need to compute all of that and put it into a single path to fill because of the need to make it appear to be a single operation. They could maybe be broken out and performed individually if the path knew that it was a bunch of lines that don't overlap, but since knowing if they overlap involves knowing the stroke width, and since the path does not store the stroke width (it is in the Paint object), then such a revelation would require a lot of computation each time the path-of-lines is drawn - so you are back to lots of computation and thus the solution is to just go through the work of computing a stroke-path and filling it.

Is there a possible optimisation to be done by exposing a drawLines option (i.e one instruction, lots of points, less geometry to be calculated, if you don't need fill or any of the other parts flar suggests)?

The drawPoints method is a single call, I'm not sure how that doesn't fit the bill here?

[ibrierley]

Ah interesting again, thank you very much. I hadn't even spotted drawPoints & PointMode.lines, I had naively assumed it was simply as it sounded, for points, so it may indeed to what I wanted faster (makes me wonder if there should be an alias drawLines for that now, I wonder how many others didn't spot that, but don't want to sidetrack the issue). I'll give it a go when I get chance in a few days and feed back anything useful I find. Thanks again.

…

On Thu, Jul 22, 2021 at 7:32 PM Jim Graham ***@***.***> wrote: I'm a little confused though...why does a drawPath have stroke geomoetry to be calculated but multiple drawLine doesn't (if that is the case and assuming here we're talking about multiple lines and not curves), is this just because of corner joins ? The geometry of a single drawLine can be easily hardcoded. If the endcaps are BUTT or SQUARE then it is a simple rectangle aligned with the direction of the line. If the endcaps are ROUND, then it is a rounded rectangle aligned along the direction of the line. A few instructions just dispatches the appropriate primitive rendering. The geometry of 2 lines in the same path invokes a general algorithm which follows the path, computes joins and caps which involves a bit of trigonometry at each vertex for the joins, etc. It also expresses the outline of all the geometry in the path in a single resulting path so that it is filled in one single operation rather than in a number of separate operations so that there is no overdraw of any of the pixels resulting in the issues described above. Even if the path could identify "hey, I'm actually just a list of moveTo/lineTo pairs", that wouldn't be enough to alleviate the need to compute all of that and put it into a single path to fill because of the need to make it appear to be a single operation. They could maybe be broken out and performed individually if the path knew that it was a bunch of lines that don't overlap, but since knowing if they overlap involves knowing the stroke width, and since the path does not store the stroke width (it is in the Paint object), then such a revelation would require a lot of computation each time the path-of-lines is drawn - so you are back to lots of computation and thus the solution is to just go through the work of computing a stroke-path and filling it. Is there a possible optimisation to be done by exposing a drawLines option (i.e one instruction, lots of points, less geometry to be calculated, if you don't need fill or any of the other parts flar suggests)? The drawPoints method is a single call, I'm not sure how that doesn't fit the bill here? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78543 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5YN5N7GXG7POQFPDQFJ5TTZBP4FANCNFSM4ZM2NDYA> .

[flar]

To be clear, drawPoints has 3 modes:

• each point is its own thing to draw, a square or a circle depending on the caps in the Paint object
• each pair of points is a line, each drawn with the caps in the Paint object separately and individually
• all of the points form a(n unclosed) polygon but the segments of the polygon are drawn separately and individually

line and polygon mode differ only in how they pair the lines up. lines takes 2 points per line and polygon takes 1 and reuses the previous point for its lines

[ibrierley]

Had a bit of a play, it seems a little faster to draw, but it also seems to crash sometimes when there's a lot of points, (with no error). Will post back if I can isolate it further.

[ibrierley]

For what it's worth I've tracked it down to

W/Adreno-GSL( 8022): <sharedmem_gpuobj_alloc:2713>: sharedmem_gpumem_alloc: mmap failed errno 12 Out of memory
E/Adreno-GSL( 8022): <gsl_memory_alloc_pure:2297>: GSL MEM ERROR: kgsl_sharedmem_alloc ioctl failed.

I can use over 5x as much memory (checking using the profiler) using drawPath and it doesn't crash though, so not sure if there's some limitation with the drawPoints method and how much memory can be allocated to the gpu or something ? Not quite sure how it all ties together. I feel this is maybe a distraction to the original problem though, so I'll leave it there.

[hcanyz]

Is there any progress now? I also encountered a similar problem.
#118434

[dnfield]

@flar did you happen to look at this on Impeller? I'd expect it to not be as problematic there but I'm not sure I'm getting the full scope of the problem.

[flar]

@flar did you happen to look at this on Impeller? I'd expect it to not be as problematic there but I'm not sure I'm getting the full scope of the problem.

I have not.

[WasteOfO2]

Does this have to do anything with hardware acceleration? On android atleast I can confirm this.

Notice how Saber doesn't use GPU

Android does have support for this, not 100% sure about on Flutter.

Android wiki for hardware acceleration

Flutter wiki for rendering performance

To test my theory, I did the following:

1. Forced 4X MSAA in dev options --> Hardware Accelerated Rendering
2. Disbaled HW overlays (forces GPU to composite screens) in dev options --> Hardware Accelerated Rendering

Now, to check whether Saber actually uses hardware rendering, I did:

1. Show View Updates in dev options

This option starts flashing all the windows that use GPU rendering in red colour (See videos)

screen-20230315-184613.mp4

screen-20230315-185008.mp4

Note: The lag may be exaggerated as I am screen recording, to be as representable of the situation to the issue as possible, I did not enable pointer location.

For reference, I also compared with the browser to check whether this is Saber specific.

So with this I can actually conclude that Saber uses software rendering, which is not efficient.

Mobile devices are affected disproportionately, as they are lower power compared to desktop/laptop counterparts.

I cannot confirm whether this issue exists on desktop OSs.

I am not 100% sure whether this is the actual culprit, but this is my best guess as for what actually be taking place.

Here, Saber is a flutter application.

This is referenced from an issue here saber-notes/saber#472

Device: Samsung Galaxy Tab S6 Lite.
OS: Lineage OS 19.1

[hcanyz]

any progress on this issue 👀

[Iey4iej3]

Or any workaround to reduce this latency?

[m0byn]

It is impacting performance strongly! In fact, some apps are barely useable. So, resolving this issue is of high priority I would argue!

[flar]

All work on path rendering is being done on the Impeller back end so it would be good to know how the performance is for your applications on that back end.

https://docs.flutter.dev/perf/impeller

[Iey4iej3]

All work on path rendering is being done on the Impeller back end so it would be good to know how the performance is for your applications on that back end.

https://docs.flutter.dev/perf/impeller

Any workaround to test it on Android? cf. saber-notes/saber#179 (comment)

[okmanideep]

The issue might not be limited to beziers.

We used the following CustomClipper<Path> with a ClipPath. It draws 80 lines to form a clip path.

class StickerClipper extends CustomClipper<Path> {
  const StickerClipper();

  static const nSpokes = 40;

  @override
  Path getClip(Size size) {
    final path = Path();
    final outerRadius = size.width / 2;
    final innerRadius = outerRadius * 0.95;
    const spokeAngle = 2 * pi / nSpokes;

    final startX = outerRadius;
    final startY = outerRadius - innerRadius;
    path.moveTo(startX, startY);
    for (var i = 0; i < nSpokes; i++) {
      final outerSpokeAngle = i * spokeAngle + spokeAngle / 2;
      final innerSpokeAngle = outerSpokeAngle + spokeAngle / 2;
      final outerSpokeX = outerRadius + outerRadius * sin(outerSpokeAngle);
      final outerSpokeY = outerRadius - outerRadius * cos(outerSpokeAngle);
      final innerSpokeX = outerRadius + innerRadius * sin(innerSpokeAngle);
      final innerSpokeY = outerRadius - innerRadius * cos(innerSpokeAngle);

      path.lineTo(outerSpokeX, outerSpokeY);
      path.lineTo(innerSpokeX, innerSpokeY);
    }

    path.close();
    return path;
  }

  @override
  bool shouldReclip(covariant CustomClipper<Path> oldClipper) => false;
}

When we put it without any transformation, it's working perfectly fine.

But when we put this inside a Pager > FittedBox > SizedBox > .
there is a huge performance issue. The debug app gets stuck when swiping to the page with this Sticker Shape.

We have other shapes with <10 lines / curves which are working perfectly fine in the exact same setup.

Will see if I can create an isolated sample to reproduce.

[ibrierley]

The issue might not be limited to beziers.

We used the following CustomClipper<Path> with a ClipPath. It draws 80 lines to form a clip path.

class StickerClipper extends CustomClipper<Path> {
  const StickerClipper();

  static const nSpokes = 40;

  @override
  Path getClip(Size size) {
    final path = Path();
    final outerRadius = size.width / 2;
    final innerRadius = outerRadius * 0.95;
    const spokeAngle = 2 * pi / nSpokes;

    final startX = outerRadius;
    final startY = outerRadius - innerRadius;
    path.moveTo(startX, startY);
    for (var i = 0; i < nSpokes; i++) {
      final outerSpokeAngle = i * spokeAngle + spokeAngle / 2;
      final innerSpokeAngle = outerSpokeAngle + spokeAngle / 2;
      final outerSpokeX = outerRadius + outerRadius * sin(outerSpokeAngle);
      final outerSpokeY = outerRadius - outerRadius * cos(outerSpokeAngle);
      final innerSpokeX = outerRadius + innerRadius * sin(innerSpokeAngle);
      final innerSpokeY = outerRadius - innerRadius * cos(innerSpokeAngle);

      path.lineTo(outerSpokeX, outerSpokeY);
      path.lineTo(innerSpokeX, innerSpokeY);
    }

    path.close();
    return path;
  }

  @override
  bool shouldReclip(covariant CustomClipper<Path> oldClipper) => false;
}

When we put it without any transformation, it's working perfectly fine.

But when we put this inside a Pager > FittedBox > SizedBox > . there is a huge performance issue. The debug app gets stuck when swiping to the page with this Sticker Shape.

We have other shapes with <10 lines / curves which are working perfectly fine in the exact same setup.

Will see if I can create an isolated sample to reproduce.

I think it maybe anything with a strokeWidth < 1, have you tried (just to test > 1?) (see earlier discussion if so)

[okmanideep]

Update on #78543 (comment)

The issue was not with the clipper but with the canvas.drawShadow() we were doing with the clipper's path. The issue went away, after removing the custom shape shadow, and having a normal circular container box shadow.

[hcanyz]

It is impacting performance strongly! In fact, some apps are barely useable. So, resolving this issue is of high priority I would argue!

any progress on this issue 👀

[theniceboy]

This issue seemed to have gotten worse on Impeller. Drawing a lot of strokes on a CustomPaint is way slower with Impeller.

[ibrierley]

This issue seemed to have gotten worse on Impeller. Drawing a lot of strokes on a CustomPaint is way slower with Impeller.

Does it make any difference if they are stroke width over size 1px vs under 1px?

[hcanyz]

@flar

Any plans on this issue? This is a devastating blow to the drawing board application.

And another way is also restricted, such as rendering ui in a non-main thread: #10647

Why is there such a problem? Is it impossible to solve it? There seems to be no similar situation on android ios native.

[flar]

Any plans on this issue? This is a devastating blow to the drawing board application.

All future performance work is targeting Impeller.

Why is there such a problem? Is it impossible to solve it? There seems to be no similar situation on android ios native.

It is not impossible to solve, but it is much easier to solve when Flutter can control the entire stack down to the hardware which is why we are focusing on Impeller for the future. You mentioned "ios native" above. Have you tried this with Flutter on iOS using Impeller?

[hcanyz]

It is not impossible to solve, but it is much easier to solve when Flutter can control the entire stack down to the hardware which is why we are focusing on Impeller for the future. You mentioned "ios native" above. Have you tried this with Flutter on iOS using Impeller?

Yes, I also tried it on iOS and had this problem too. I can assist with testing, is there anything I can provide?

flutter doctor -v
[✓] Flutter (Channel stable, 3.13.1, on macOS 13.5 22G74 darwin-x64, locale
    zh-Hans-CN)
    • Flutter version 3.13.1 on channel stable at ***
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision e1e47221e8 (2 days ago), 2023-08-22 21:43:18 -0700
    • Engine revision b20183e040
    • Dart version 3.1.0
    • DevTools version 2.25.0
    • Pub download mirror https://pub.flutter-io.cn
    • Flutter download mirror https://storage.flutter-io.cn

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at ***
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /Applications/Android
      Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      17.0.6+0-17.0.6b829.9-10027231)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E300c
    • CocoaPods version 1.12.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build
      17.0.6+0-17.0.6b829.9-10027231)

[✓] Connected device (3 available)
    • iPhone 14 Pro Max (mobile) • 12492226-C0F8-41CD-B67C-728E9063008B • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
    • macOS (desktop)            • macos                                •
      darwin-x64     • macOS 13.5 22G74 darwin-x64
    • Chrome (web)               • chrome                               •
      web-javascript • Google Chrome 116.0.5845.110

[✓] Network resources
    • All expected network resources are available.

• No issues found!

[PKiman]

I tested Flutter SDK 3.16.9 with impeller enabled on iOS.
In our test we use lots of strokes rendered with quadraticBezierTo and drawPath.
The performance is not better compared to 3.13.9 skia engine.
We tested the app with debug and release flag.
Scrolling the canvas drops frames.