*save gets really slow after composite-ing lots of images

[chregu]

Strange behaviour here, hopefully there's an easy explanation.

I load each frame of a animated gif as a vips image. Then a composite them on each other from start to end to replace the transparency with the colours from the frames before. and then I save them. But that save gets slower and slower with each image. When I don't do the composite, it's as fast as expected. The more frames it has the slower it gets.

Here's some example code which shows the behaviour:

$image = \Jcupitt\Vips\Image::newFromFile("liip-blog-animated.gif", ['n' => -1]);

$pageHeight = $image->get("page-height");
$height = $image->get("height");
$pages = $height / $pageHeight;
$images = [];
// extract toiletpaper image into single images per frame
for ($i = 0; $i < $pages; $i++) {
    $images[] = $image->extract_area(0, $i * $pageHeight, $image->width, $pageHeight);
}
$image = null;
$before = null;

foreach($images as $key => $image) {
    //put one picture on another to have the gif animation transparency replaced with the color from before
    if ($before) {
         $images[$key] = $image->composite([$before,$image],[2]);
    }
    $before = $images[$key];
}

// save them individually, doesn't matter if tiffsave_buffer or tiffsave, or pngsave
foreach($images as $key => $image) {
    $start = microtime(true);
    $foo = $image->tiffsave_buffer( ['compression'=> \Jcupitt\Vips\ForeignTiffCompression::NONE]);
    print "Run $key took " . round((microtime(true) - $start) * 1000, 1) ." ms \n";
}

An animated gif with lots of frames where it's especially noticeable can be found here https://liip.rokka.io/www_raw/170a46a35209b1747256224e037bceaab2049a51/animation-1.jpg

[chregu]

ah, and the result on my machine (virtualized) is:

Run 0 took 4.8 ms
Run 1 took 15.7 ms
Run 2 took 25.4 ms
Run 3 took 31.5 ms
Run 4 took 40.9 ms
Run 5 took 47.9 ms
Run 6 took 58.7 ms
Run 7 took 67.7 ms
Run 8 took 76 ms
Run 9 took 83.2 ms
Run 10 took 90.9 ms
Run 11 took 101.2 ms

[snip, it almost linearly increases by each run]

Run 74 took 678.5 ms
Run 75 took 681.3 ms
Run 76 took 684.1 ms
Run 77 took 696.9 ms
Run 78 took 714.8 ms
Run 79 took 719.2 ms
Run 80 took 735.1 ms
Run 81 took 732.9 ms
Run 82 took 806.9 ms
Run 83 took 751.1 ms
Run 84 took 763.1 ms
Run 85 took 774 ms
Run 86 took 774.3 ms
Run 87 took 774 ms
Run 88 took 788.2 ms
Run 89 took 807.3 ms
Run 90 took 800.1 ms
Run 91 took 812.5 ms
Run 92 took 815.6 ms
Run 93 took 816.8 ms
Run 94 took 811.4 ms
Run 95 took 856.7 ms

[jcupitt]

Hello, that's right, as the pipeline gets longer it'll run more slowly, since it's joining more images.

Most libvips images are not real images, they are just tiny scraps of code and data that get joined together. This is why it uses little memory and can process huge images.

When you execute:

$x = Vips\Image::newFromFile("x.jpg");
$x = $x->invert();

It's not actually processing any pixels. The load will check the image header and set up things like with and height, but it will not decompress anything, just attach a function to $x which will load pixels when they are needed. Likewise, invert() will not process anything, it will just add another set of functions to the image which will calculate the invert when pixels are needed.

When you finally execute:

$x->writeToBuffer(".jpg");

The final write will pull pixels through the pipeline in (usually) 128 x 128 pixels chunks. The decode, invert and recode will all execute at the same time and the work will be spread over your available cores.

In your case, you want to reuse results from one pipeline computation (the save) in the next (the next composite). You need to keep a copy of the complete intermediate image in memory. The simplest way is to use ::copyMemory() at each stage, something like:

$base = $frame[0];
for ($i = 1; i < $n_frames; $i++) { 
    // this will make a new pipeline section from base, but will not do any computation
    $base = $base->composite($frame[$i], 2);
    // this will allocate a new area of memory, then render the image into it
    // the previous memory image will be GCd away soon, hopefully
    $base = $base->copyMemory();
    // save this intermediate
    $my_image_string = $base->writeToBuffer(".tif");
}

https://jcupitt.github.io/php-vips/docs/classes/Jcupitt.Vips.Image.html#method_copyMemory

Stepping back, I'm not certain that this is necessary. Each frame of the gif loader output is already a composite of every previous frame, so I think you can just pull single frames out and save them. Or I'm missing something!

[jcupitt]

More background:

libvips has quite a lot of machinery to try to cache and reuse results within pipelines. For example, suppose you have something like:

$a = $a->invert();
$r, $g, $b = $a->bandsplit();
$r += $g;
$b = $r->bandjoin([$g, $b]);
$b->writeToFile(...);

(not legal php, I know)

ie. a pipeline that splits an image up, processes, and rejoins, you might think there's a danger that the invert could execute three times for each pixel as each of $r, $g and $b are computed. In fact libvips is able to spot the reuse of $a in the small per-thread cache of recently calculated pixel buffers and skip the second and third computations.

It starts to get harder when there are area operations, like convolution. libvips tries to optimise these cases by reordering computation so that the largest areas are computed first and there's the most likelihood of finding a buffer to reuse.

Operations with small value results, like image average or image histogram, are memoised. Before adding any operation to a pipeline, libvips searches the last 1,000 operations and simply reuses the result of the previous invocation. This is safe, because libvips images are immutable.

This has the extra effect of common-subexpression elimination. If you have:

$c = $a->add(2);
$b = $b->add($a->add(2))->add($c);

The $a->add(2) will only be computed once.

Finally, you can add explicit cache nodes to a pipeline. tilecache and friends can save chunks of image and share them between threads. They do slow things down though, so you need to avoid overuse.

http://jcupitt.github.io/libvips/API/current/libvips-conversion.html#vips-tilecache

https://jcupitt.github.io/php-vips/docs/classes/Jcupitt.Vips.ImageAutodoc.html#method_tilecache

[chregu]

ah thanks, that explains a lot. And copyMemory makes it fast.

About gif loading and composition. the single pages still have transparency in it, which is noticeable when you resize just on that.

With the image from http://files.chregu.tv/animated.gif

$image = \Jcupitt\Vips\Image::newFromFile("animated.gif", ['n' => -1]);
$image->pngsave("flat.png");

Ends up in

composite helps removing them.

[jcupitt]

But won't composite break things like a solid object moving over a transparent background? You'll make each frame the superposition of all the previous object positions.

There's something I'm not understanding about gif :(

[chregu]

I'm not an gif expert either, for me, transparent just meant "take the same color as the frame before on that frame" and one should apply that during optimizing it. But I don't know exactly, the end result should be the same

The problem is when I don't do composition with the background and just for example resize on such an image, one can see later the artifacts where it was transparent, since it antialiases between the transparency and eg. the blue full color (in the above example). If I composite the frame before, that doesn't happen.

[jcupitt]

Hmm I think I found it:

http://www.webreference.com/content/studio/disposal.html

libvips isn't implementing the dispose stuff at the moment. I'll try to add it.

I added a metadata item for gif-comment, by the way.

[jcupitt]

It seems to handle dispose correctly now, or at any rate all my test gifs seem to work.

You shouldn't need the composite thing any more: just load with n=-1 and each 'sheet' will be the complete image for that frame.

Thank you for pushing on this -- it's good to fix up the gif loader.

[jcupitt]

I'll close, this should be fixed in 8.6.