[QUESTION] How to copy one bitmap into another with offset?

[Jan1503]

Hi!

I'm new to Skia and try to use it for some pixel-manipulations on my LED-Matrix on a Raspberry Pi.

I would like to know if there's an easy way to copy one (smaller) bitmap into another (larger) bitmap at a given location.

So lets say I have two bitmaps:

var entireBitmap = new SKBitmap(64, 32); var smallBitmap = new SKBitmap(16,8);

Now I want to copy the entire 'smallBitmap' into the 'entireBitmap' at let's say x = 12, y= 18.

I think I could do it using GetPixel and SetPixel but thats slow and needs some calculations so I think maybe theres a better way?!

It should also accept negative location coordinates and tolerate overlapping if the smaller bitmap goes beyond the height or width of the entire bitmap.

Someting like SKBitmap.CopyTo(entireBitmap, smallBitmap, x, y);

Thanks and regards
Jan

[Jan1503]

Hmm, it looks like I can draw a bitmap to a bitmap with an offset using a canvas like this:

static void CopyWithOffset(SKBitmap targetBitmap, SKBitmap sourceBitmap, SKPoint targetLocation) { using var canvas = new SKCanvas(targetBitmap); canvas.DrawBitmap(sourceBitmap, targetLocation); canvas.Flush(); }

Will test tomorrow but looks good :)

[Jan1503]

It works as expected but I don't know if this is the right way to go...
If I have many threads, each one will copy it's smaller bitmap into the larger one on a different location.
As SKBitmap is surely not thread-safe this might end up in a pixel-mess, especially if some bitmaps overlap others.

What would be the best way to draw multiple bitmaps into a big one without messing up the main bitmap?
I could maybe lock the big bitmap, write the small ones one by one and release the big one. But this smells to be a slow process which might led to flicker on my led-screen.

Any suggestions?

[ziriax]

If I have many threads, each one will copy it's smaller bitmap into the larger one on a different location.

Although Skia is not thread safe, if the destination rectangles don't overlap, you could try to make different destination SKBitmap and SKCanvas instances per thread, but all using the same destination memory. You can use SKBitmap.InstallPixels for that. Note that in that case the SKBitmap does not take ownership of the pixel buffer.

If the rectangles do overlap, using multiple threads won't work of course...

Out of curiosity, why are you using multiple threads for a single drawing?

[Jan1503]

Out of curiosity, why are you using multiple threads for a single drawing?

I have a big LED-Screen (384x192) which is fed using https://github.com/hzeller/rpi-rgb-led-matrix.
Its using a canvas where I can set each pixel using a simple 'canvas.SetPixel' method. After "drawing" to that canvas one must call 'matrix.SwapOnVsync' to display the canvas on the screen.

Thats all easy as long as only one Task writes to the canvas. But I want to have multiple "layers" to be able to draw different things to different parts of the canvas, for example a scrolltext at 64,32 (x,y), an image to 128,96 and so on.
So I created different canvi with different dimensions and locations on the big canvas. Each canvas represents an SKBitmap and so does the big canvas.

In the main thread, before pushing out the main bitmap to the canvas, I call a method on each "child-canvas' that stops the writing to the corresponding child-canvas, copies the bitmap to the main-bitmap and then releases the lock so the child-canvi could be updated again.

This all works but results in heavy flicker and 'jumping' text, as the RPi4 semms to be not fast enough to lock all Tasks that write to the child-bitmap, coyping the child-bitmap to the big bitmap and pushing it out to the LED-Display.

I've experimented with some Thread.Sleeps and so on, but thats all crap...Even going one ladder down and using Threads with ManualResetEvents doesn't work well.

So I need an ultrafast solution that allows me to write to my child-bitmaps while the 'main' bitmap is constantly pushed out the LED-Matrix.
I tried using the old System.Drawing.Bitmap and lock/unlock the bits in unsafe blocks, but thats even slower than using 'SKBitmap.GetPixels()' so I discarded that approach.

I think you got the point but for completeness here's a small snipped from my main-thread:

        private void ThreadImplementation()
        {
            while (true)
            {
                _mre.Reset();
                foreach (var canvas in _canvi)
                    canvas.PrepareNextFrame();
                _mre.Set(); 

                DrawBitmapPixelsToCanvas(_ledMatrix, _mainCanvas, _mainCanvasBitmap);
                //DrawBitmapPixelsToCanvasOnlyChangedPixels(_ledMatrix, _mainCanvas, _mainCanvasBitmap);
            }
        }

And the DrawPixels-Method:

        private static void DrawBitmapPixelsToCanvas(RGBLedMatrix ledMatrix, RGBLedCanvas canvas, SKBitmap bitmap)
        {
            var pixelAddr = bitmap.GetPixels();
            unsafe
            {
                var ptr = (byte*)pixelAddr.ToPointer();

                for (var col = 0; col < bitmap.Height; col++)
                for (var row = 0; row < bitmap.Width; row++)
                {
                    var b = *ptr++;
                    var g = *ptr++;
                    var r = *ptr++;
                    ptr++; //Skip alpha

                    canvas.SetPixel(row, col, new RGBLedMatrixWrapper.Color(r, g, b));
                }
            }
            
            ledMatrix.SwapOnVsync(canvas);
        }

And here's the method in the child-canvas class that is called from the '...PrepareNextFrame()'-method:

    private static void CopyWithOffset(SKBitmap targetBitmap, SKBitmap sourceBitmap, SKPoint targetLocation)
    {
        using var canvas = new SKCanvas(targetBitmap);
        canvas.DrawBitmap(sourceBitmap, targetLocation);
    }

targetBitmap represents the main 'big' bitmap and sourceBitmap is the smaller child-bitmap that should be written 'into' the targetBitmap at a specific location.

Although Skia is not thread safe, if the destination rectangles don't overlap, you could try to make different destination SKBitmap and SKCanvas instances per thread, but all using the same destination memory. You can use SKBitmap.InstallPixels for that. Note that in that case the SKBitmap does not take ownership of the pixel buffer.

Could you give me an example on how to implement that?

Thanks!

[ziriax]

I don't get it, 384x192 is really small, modern CPUs should be able to handle that in real time on a single thread, even with multiple layers being composed on top. Not sure about a PI4, but we used to do that on old Acorn Archimedes and i386 computers in the 90's, so I guess the modern PI should be able to do that too.

Did you try to do some profiling where the bottleneck is? I wouldn't be surprised that the bottleneck is actually the many SetPixel calls, C# -> C++ marshalling overhead. You might be able to improve that with C# 9 unmanaged delegates, but that would require modifying the C# wrapper code. But first do some profiling.

So my advice is, start with just one thread, a single SKCanvas and single backing SKBitmap, and profile what is slow.

• if it turns out SetPixel is slow, modify the wrapper to give an unmanaged delegate to the SetPixel call, or even replace all the C++ code that converts the chunky Skia pixels to the planar LED bit-planes to unsafe C# code.
• if SetPixel is slow, and you can't modify the C# wrapper, try using just two threads, with two double buffered SKCanvas and SKBitmap objects. One thread is doing Skia rendering, the other thread is converting the pixels with SetPixel (DrawBitmapPixelsToCanvas)
• in the unlikely case that Skia rendering is slow on PI4, then you need to first profile what parts are slow.

In general, always first measure what is slow.

If you can't use a real profiler, just using a C# StopWatch might be good enough to identify the slow parts.

This mostly feels non-SkiaSharp related. If this is a commercial project and you need urgent help, then please reach out to me privately, I might be able to find people to help you out.

PS: In the code of the LED API I also found the following regarding the "flicker"

bool RGBMatrix::Impl::StartRefresh() {
  if (updater_ == NULL && io_ != NULL) {
    updater_ = new UpdateThread(io_, active_, params_.pwm_dither_bits,
                                params_.show_refresh_rate,
                                params_.limit_refresh_rate_hz);
    // If we have multiple processors, the kernel
    // jumps around between these, creating some global flicker.
    // So let's tie it to the last CPU available.
    // The Raspberry Pi2 has 4 cores, our attempt to bind it to
    //   core #3 will succeed.
    // The Raspberry Pi1 only has one core, so this affinity
    //   call will simply fail and we keep using the only core.
    updater_->Start(99, (1<<3));  // Prio: high. Also: put on last CPU.
  }
  return updater_ != NULL;
}

[Jan1503]

I don't get it, 384x192 is really small

You're right, it IS small but according to the folks using this library it's the highest resolution a single Pi4 can handle with acceptable refresh rate. I built it using 3 chains of 3 128x64 modules. I also set https://github.com/hzeller/rpi-rgb-led-matrix#cpu-use so that the refresh takes place on a dedicated cpu.

I wouldn't be surprised that the bottleneck is actually the many SetPixel calls, C# -> C++ marshalling overhead.

That's a discussion I have read many times in the issue-section of the LED-Repo.

You might be able to improve that with C# 9 unmanaged delegates, but that would require modifying the C# wrapper code.

I have access to the bindings as they're also in the repo at https://github.com/hzeller/rpi-rgb-led-matrix/tree/master/bindings/c%23 but I'm not skilled enough to rewrite it using unmanaged delegates.

So my advice is, start with just one thread, a single SKCanvas and single backing SKBitmap, and profile what is slow.

If I throw away all my canvas-logic, even all Skia-related stuff and write directly to the led-matrix using canvas.SetPixel it's blazing fast.
It also 'looks good' if I use static images or text so I guess a bottleneck might be in the way im trying to perform scrolling text.
Look at my method here to get the idea:

public Task DrawScrollText(string text, int xPos, int yPos, int delay, SKColor color, int loops = -1,
        CancellationToken ct = default)
    {
        return Task.Run(() =>
        {
            var infinite = loops == -1;

            var typeFace = SKTypeface.FromFamilyName("Comic Sans MS", SKFontStyleWeight.Thin,
                SKFontStyleWidth.Normal, SKFontStyleSlant.Oblique);
            using var textPaint = new SKPaint
            {
                TextSize = 24.0f,
                IsAntialias = true,
                Color = color,
                Typeface = typeFace,
                FilterQuality = SKFilterQuality.High,
                TextEncoding = SKTextEncoding.Utf16
            };

            var textWidth = (int) Math.Ceiling(textPaint.MeasureText(text));
            using var skCanvas = new SKCanvas(_myBitmap);
            while (infinite || loops >= 0)
            {
                loops--;

                for (var i = _myBitmap.Width; i >= -textWidth; i--)
                {
                    _mre.WaitOne();
                    //skCanvas.Clear(SKColors.Black);
                    _myBitmap.Erase(SKColors.Black); //<-- This seems to be really inefficient.
                    skCanvas.DrawText(text, i, yPos, textPaint);
                    Thread.Sleep(delay);

                    if (ct.IsCancellationRequested)
                    {
                        Console.WriteLine("DrawScrollText-Task canceled.");
                        throw new OperationCanceledException(ct);
                    }
                }
            }
        }, ct);
    }

I need to clear the canvas/bitmap before writing the new text on the next position. This might slow down things and looks inefficient but I don't know another way...

If this is a commercial project and you need urgent help...

No, I like all glowing things, espcially LEDs...its all just for fun and this problem bugs me a lot :) It's the first time I use Skia as I think it could be working well for this.

Thanks for your help!

[ziriax]

I was assuming he is not doing that, the loop itself doesn't at least.

[Gillibald]

I assumed he calls this multiple times could be wrong

[Jan1503]

Guys, this is all some experimenting and will be refactored once the 'overall' design works as expected.
As @ziriax said I'm not doing it in the "scrolling"-loop so it's not part of my current problem.

However I took another approach and pre-rendered the text to a bitmap and scroll the bitmap instead of the text.
Guess what, it's the exact same behavior.

public void DrawScrollText()
    {
        var text = "Hinter eines Baumes Rinde saß die Made mit dem Kinde, denn der Gatte, den sie hatte, fiel vom Blatte und diente auf diese Weise einer Ameise als Speise...";
        var loops = -1;
        var color = new SKColor(255, 255, 255);
        var yPos = 18;
        var delay = 15;

        var typeFace = SKTypeface.FromFamilyName("Comic Sans MS", SKFontStyleWeight.Thin,
            SKFontStyleWidth.Normal, SKFontStyleSlant.Oblique);
        using var textPaint = new SKPaint
        {
            TextSize = 24.0f,
            IsAntialias = true,
            Color = color,
            Typeface = typeFace,
            FilterQuality = SKFilterQuality.High,
            TextEncoding = SKTextEncoding.Utf16
        };

        var textWidth = (int)Math.Ceiling(textPaint.MeasureText(text));
        using var skCanvas = new SKCanvas(_myBitmap);
        
        var textBitmap = new SKBitmap(textWidth, (int) textPaint.TextSize);
        using var textCanvas = new SKCanvas(textBitmap);
        textCanvas.Clear(SKColors.Black);
        textCanvas.DrawText(text,0,yPos,textPaint);
        
        //return Task.Run(() =>
        while (true)
        {
            var infinite = loops == -1;
            
            while (infinite || loops >= 0)
            {
                loops--;

                for (var i = _myBitmap.Width; i >= -textBitmap.Width; i--)
                {
                    _mre.WaitOne();
                    skCanvas.Clear(SKColors.Black);
                    //_myBitmap.Erase(SKColors.Black); //<-- This seems to be really inefficient.
                    //skCanvas.DrawText(text, i, yPos, textPaint);
                    skCanvas.DrawBitmap(textBitmap,new SKPoint(i,0));
                    Thread.Sleep(delay);
                }
            }
        }//, ct);
    }

[Jan1503]

It even makes no difference if I'm using the ManualResetEvent or not. I'm using exactly 2 real Threads here, one for the overall Bitmap which is pushed to the LED-Matrix and another one that runs the above Method.
The "basic" scrolling works, but it is not smooth and stucks repeatedly for a microsecond at different positions so it seems there might be some background processes that interfere with my program?!
I'm no expert in the underlying OS (Raspbian) so I have only taken the optimizations from HZellers site.

[Jan1503]

Here's a short video showing the aforementioned method in action. The 'hickups' are barely visible on the video but clearly in reality.

IMG_4918.mp4

[ziriax]

You seem to be mixing two different threading models. You use Task.Run, and Thread.Sleep(delay);. Task.Run is for lightweight tasks that might even run on a single thread, unless you give the hint "LongRunning". I guess you really intended to create a dedicated thread for this? I suggest to replace Task.Run with a real Thread. You might also want to play with the thread affinity.

Also, I would not use a Thread.Sleep at all. You want to be synchronized to the LED's display refresh rate. Just use event handles like _mre.WaitOne(); for that, and remove the sleep. But you need bidirectional communication. You can use a queue for that, or just a good old semaphore.

For the Skia bits, it might indeed be the case that rendering text is slow on a PI. I would put a StopWatch around it and check.

Or, you could start with a simple moving solid color square on a solid color background, and also test changing all the colors of all the pixels every frame, background and foreground, because the SetPixel might have an internal optimization that it will not change the LED when the same color is passed to it (not sure about that, just guessing). If that is fast, then text rendering is slow. Now on such a small display I wouldn't use real fonts anyway, I would you a single bitmap with handmade bitmap fonts, and take tiles out of that one. Or pre-render the glyph to a bitmap, enabling hinting, and then use that bitmap. Do a google image search for font sprite sheet for example.

[Jan1503]

Also, I would not use a Thread.Sleep at all. You want to be synchronized to the LED's display refresh rate. Just use event handles like _mre.WaitOne(); for that, and remove the sleep.

The Thread.Sleep is for the loop to specify the pause = scrolling speed before the text moves to the next position. If I remove it the text 'rushes' thru the display and is not readable. I don't know how to accomplish this any other way.

I guess you really intended to create a dedicated thread for this? I suggest to replace Task.Run with a real Thread.

You're right, I've experimented with Tasks and Threads, knowing that many Tasks may run on a single Thread but even after changing my Drawing-Methods to use explicit Threads instead of Tasks makes no relevance.

As it turns out this is too much advanced stuff for me so I should just stick with a simple blinking LED-Example ;(

[ziriax]

Well, it's good fun 😄 Good luck anyway