Canvas operations are extremely slow

[creativecreatorormaybenot]

When I try to use a loop that draws multiple hundreds of thousand pixels using Canvas.drawRect, I get render times of multiple seconds. Why is that and how should I draw pixels instead?

I elaborated on this problem on StackOverflow: https://stackoverflow.com/q/55844043/6509751

[daniilguit]

Haven't you tried Canvas.drawRawPoints?
https://docs.flutter.io/flutter/dart-ui/Canvas/drawRawPoints.html

[creativecreatorormaybenot]

I can answer it myself: Canvas.drawRect is not meant to be called this often. I solved it using Canvas.drawImage and the BMP file format.

[andrewackerman]

@creativecreatorormaybenot The solution listed in the StackOverflow answer doesn't work for realtime effects. It needs Futures/callbacks that require at least an additional frame before the Image object can be rendered to the canvas. Flutter needs a synchronous way to create an Image object, or a way without needing the Image class to render raw pixel data.

@daniilguit ...and no, it's not drawRawPoints, since that just draws the same paint to a number of points defined by a list of Offset. The idea is to supply the canvas with raw pixel data, like how in the StackOverflow solution to render a BMP, but without relying on the asynchronous Image interface.

[creativecreatorormaybenot]

@andrewackerman I do not think that the asynchronicity of the solution I proposed is a problem, let me explain.

In my case, I can draw at 60 FPS up to a certain resolution. I will surely have to do same optimizations going forward and I might post them here if I get reminded.
This means that it might take a frame to render the image, however, it would take one frame anyway, even if it was synchronous. I simply convert to BMP before I trigger the paint call, which should yield the same end result as triggering the paint call and then creating the image synchronously in the paint call.

If your concern is about performance, then I am with you. Currently, I am having memory issues in particular, however, I believe (at least I hope) that this is the fault of my implementation.
However, synchronicity would not solve the performance problem either as synchronous implementations would overload the device in the same way asynchronous calls would.

Maybe I misunderstood your point. Otherwise, I could help you find an implementation that allows real-time effects, however, I should probably optimize performance first as my method does not translate well to high resolutions.

[andrewackerman]

@creativecreatorormaybenot The nature of asynchronicity in Dart means that the rendering of a frame is deferred to the next frame. This isn't always a reasonable compromise, for example when you don't want the canvas to paint until a frame is available, or when you are running the generation on every frame possible. At best, this requires a complicated scheduling system where the element is trying to generate a frame during the painting step of the previous frame, and the future scheduler doesn't always make it clear when that frame will be available. This system also means that a frame will almost never be available during the first painting pass.

This is a lot of compromises and jumping through hoops that would be entirely avoided by the existence of a synchronous way to pass raw pixel data to the canvas.

To explain my use case, I am trying to make a plugin that renders noise to a canvas using the fast_noise library. In the simplest case, I generate greyscale pixel data from a 2D/3D Simplex Noise function. At this point, it would be ideal if I could simply pass this pixel data to the canvas for rendering. Instead, I'm forced to use decodeImageFromPixels and store the resulting image ahead of time for the next painting pass. It also forces me to dispose the previous image after it has been rendered. After all is said and done, the resulting widget sees <1 FPS. I attribute some of this to the noise generation function (it's probably a pipe dream to expect seamless 60 FPS performance from a software-implemented noise generator and particularly one that doesn't utilize SIMD instructions) but a good chunk of that performance loss comes from the asynchronicity and image object generation/disposal as well.

I haven't tried your BMP solution to see if it's any different, but between its similar asynchronous nature and the fact that decodeImageFromPixels calls _instantiateImageCodec internally, I would imagine that performance would be largely identical.

[creativecreatorormaybenot]

@andrewackerman I think that I was simply not aware of decodeImageFromPixels, which is why I went through implementing my own BMP encoder (which was a pain), however, I would suspect that the performance is very similar.

I might give it a try in the next few days and report back (maybe update the SO answer), however, I would say that the performance should really be exactly the same if I am not doing something horribly wrong in my implementation.

This is a lot of compromises and jumping through hoops that would be entirely avoided by the existence of a synchronous way to pass raw pixel data to the canvas.

You are totally correct and I would suggest that you open an issue of this, i.e. a feature request for "a synchronous way to pass raw pixel data to the canvas". It would be nice if you could reference this issue or put a link here.

[creativecreatorormaybenot]

@andrewackerman I am glad to be able to report back.

I tested decodeImageFromPixels and it is definitely more efficient than my implementation, however, that is probably more memory efficient (I am still not completely sure). However, the performance is still not awesome as you rightfully pointed out. Thus, I would still say that opening an issue is a good idea.

On another note, are you able to run your code on master?
See #36312. Basically, I am only able to run it on the beta channel.

[Solido]

@andrewackerman Did you open a ticket about pushing data directly to the canvas ?
So I can upvote it. Thx

[andrewackerman]

I've been planning on doing it, but real life stuff has been hitting me pretty hard the past few weeks. My thought was that it should have a similar signature to decodImageFromPixels, so something like:

class Canvas {
  ...

  void drawPixels(
    Offset c, 
    Uint8List bytes,
    int width,
    int height,
    PixelFormat pixelFormat, {
    int rowBytes,
    int targetWidth,
    int targetHeight,
  }) {
    assert(c != null && bytes != null && width != null && height != null && pixelFormat != null);
    assert(bytes.length == width * height * 4);
    if (rowBytes == null) rowBytes = width * 4;
    if (targetWidth == null && targetHeight == null) {
      targetWidth = width;
      targetHeight = height;
    } else if (targetWidth == null) {
      targetWidth = (targetHeight * (width / height)).toInt();
    } else if (targetHeight == null) {
      targetHeight = (targetWidth * (height / width)).toInt();
    }
    _drawPixels(c, pixelData, width, height, pixelFormat, rowBytes, targetWidth, targetHeight);
  }
}

c is the position that the image will be drawn at.

bytes is the pixel data that will be drawn to the canvas, represented as byte groups of 4 bytes per pixel. The color channels will be extrapolated based on the value of pixelFormat.

width and height are the dimensions of the image represented by the pixel data. If width * height * 4 is not equal to pixelData.length, an error will be thrown.

rowBytes is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided format.

The targetWidth and targetHeight arguments specify the size of the output image, in image pixels. If they are not equal to the intrinsic dimensions of the image, then the image will be draw. If exactly one of these two arguments is specified, then the aspect ratio will be maintained while forcing the image to match the other given dimension. If neither is specified, then the image maintains its real size. (Debatable as to whether these parameters should be supported as they don't really fall into the whole "just draw these pixels" mentality of this method.)

[creativecreatorormaybenot]

@Solido #37180.

If any of you think that something about it should be changed, just message me or comment there.