Port `vips_thumbnail` logic to sharp #2789

kleisauke · 2021-07-09T11:56:42Z

As discussed in #2787, this PR ports some vips_thumbnail logic to sharp. It implements "scale-on-load" for WebP, SVG and PDF images, which allows an image to be resized in one go without the need for an intermediate resizing step. Previously sharp uses the deprecated shrink option for webpload, so I expect WebP images to be resized a bit faster within this PR.

This PR also ports the corresponding page-height logic of vips_thumbnail, allowing resizing of animated/multipage images without having to specify the pageHeight option for the .webp() and .gif() methods (this covers issue #2275).

Before / after examples

$ curl -LO https://storage.googleapis.com/downloads.webmproject.org/webp/images/dancing_banana2.lossless.webp
$ node -e "require('sharp')('dancing_banana2.lossless.webp', { pages: -1 }).resize(128).toFile('before.gif')"
$ vipsthumbnail dancing_banana2.lossless.webp[n=-1] -s 128x -o after.gif

Before	After

Setting `.gif({ pageHeight: 136 })` doesn't work.	Setting `.gif({ pageHeight: 136 })` isn't necessary.

$ curl -LO https://github.com/lovell/sharp/raw/master/test/fixtures/circle.svg
$ node -e "require('sharp')('circle.svg').resize(1024).toFile('before.png')"
$ vipsthumbnail circle.svg -s 1024x -o after.png

Before	After

Marked this PR as a draft due to these TODO-items:

Add unit tests.
Should fastShrinkOnLoad produce the same "normalised" dimensions if it wasn't enabled?

These features of vips_thumbnail are currently missing from sharp, but might be included later:

Shrink-on-load for page-based and ifd-based TIFF pyramids.
Shrink-on-load for HEIC/AVIF thumbnails.
Shrink-on-load for page-based JP2K pyramids (requires vips 8.11).
Shrink-on-load for Openslide levels.

coveralls · 2021-07-09T11:59:58Z

Coverage remained the same at 100.0% when pulling d72f4bc on kleisauke:vips-thumbnail into add4c79 on lovell:main.

ahukkanen · 2021-07-09T16:54:55Z

@kleisauke at #2787 you mentioned:

The previous implementation returned both 320x261 with fastShrinkOnLoad enabled or disabled, vipsthumbnail returns 320x262 as dimension when patching libvips like this [...]

So taken for example this test you changed:

sharp/test/unit/resize.js

Line 26 in 82eaa8b

assert.strictEqual(262, info.height);

What happens is:

The source image is 2725x2225
Shrink-on-load is set to 8
After the shrink load, the image dimensions are 340x278 -- This no longer matches the original dimensions, when rounded correctly it would be 341x278
And this causes the final image height mismatch because it is resized proportionally to the size after the shrink-on-load, not the original dimensions

It is explained in further detail here:
https://github.com/libvips/libvips/blob/757d03100566368e5d06c7355e73c1215dd34968/libvips/foreign/jpeg2vips.c#L586-L596

cinfo->output_width and cinfo->output_height round up with
shrink-on-load. For example, if the image is 1801 pixels across and
we shrink by 4, the output will be 450.25 pixels across,
cinfo->output_width with be 451, and libjpeg will write a black
column of pixels down the right.

We must strictly round down, since we don't want fractional pixels
along the bottom and right.

So when you redefine inputWidth and inputHeight the mismatch happens here in case either dimension got rounded down:

sharp/src/pipeline.cc

Lines 239 to 241 in 82eaa8b

    
           // Any pre-shrinking may already have been done. 
        
           inputWidth = image.width(); 
        
           inputHeight = image.height();

lovell · 2021-07-14T10:03:30Z

This is amazing, thank you Kleis.

Should fastShrinkOnLoad produce the same "normalised" dimensions if it wasn't enabled?

Based on the principle of "least surprise", I'd probably say yes, if we can do so whilst making this change.

Would making the following alteration, to bring the logic in sharp more in line with libvips, help prevent some rounding problems? (I know we've previously discussed doing this, and now feels like the right time.)

- if (shrink >= 8 * factor) {
+ if (shrink >= 16 * factor) {
    jpegShrinkOnLoad = 8;

kleisauke · 2021-07-15T10:44:55Z

Thanks for the analysis @ahukkanen! I tried to implement a round-up behavior in jpeg2vips.c a while ago, but it didn't work out.

I had a go to produce "normalised" dimensions with commit 1769bc8 by recalculating vshrink / hshrink on the main image instead of the pre-shrunk image. This seems to work in most cases, but it might misbehave with the page-based TIFF pyramid shrink-on-load logic (which is not yet implemented in this PR).

Details

Tested commit: kleisauke/libvips@d7ed0dd.

$ curl -LO http://merovingio.c2rmf.cnrs.fr/iipimage/PalaisDuLouvre.tif
$ vipsthumbnail PalaisDuLouvre.tif -s 500x103
$ vipsheader tn_PalaisDuLouvre.jpg
tn_PalaisDuLouvre.jpg: 498x103 uchar, 3 bands, srgb, jpegload
$ vips copy PalaisDuLouvre.tif[page=0] x.jpg
$ vipsthumbnail x.jpg -s 500x103
$ vipsheader tn_x.jpg
tn_x.jpg: 498x103 uchar, 3 bands, srgb, jpegload

The first operation resizes the third page of the pyramid TIFF from 500x103 to 498x103 as it calculates the vshrink / hshrink on the main image. Without that commit, libvips will produce an 500x103 image.

Would making the following alteration, to bring the logic in sharp more in line with libvips, help prevent some rounding problems? (I know we've previously discussed doing this, and now feels like the right time.)
- if (shrink >= 8 * factor) {
+ if (shrink >= 16 * factor) {
    jpegShrinkOnLoad = 8;

Probably the other shrink factors will also need to be adapted to leave at least a factor of two for the final resize step (but that would deprecate fastShrinkOnLoad, so I'm not sure if that's desired).

ahukkanen · 2021-07-15T14:49:02Z

I'm not sure about the performance implications @kleisauke but couldn't it be also solved just by cutting one extra pixel off the shrinked image when this happens?

So, for the given example above, you would cut 1px off the height and the image would become 340x277 after the shrink. Using this aspect ratio, the resulting image would then become 320x261 as it originally was.

I'm also not sure if that works perfectly in every case but at least it would be closer to the current behavior, if that's the problem.

lovell · 2021-07-16T11:27:49Z

Thanks for the updates Kleis, I'm on the fence about whether to keep or remove fastShrinkOnLoad. Perhaps we should wait until libvips supports the proposed RGBX colourspace and therefore improve vips_reduce performance first?

chrisdostert · 2021-08-16T02:24:22Z

Any thoughts/updates on getting this merged? I've been using @kleisauke 's branch and it's been working like a charm. Thanks for everyones hard work and contributions on this wonderful project!

kleisauke · 2021-08-27T10:53:58Z

My apologies for the delay.

I'm not sure about the performance implications @kleisauke but couldn't it be also solved just by cutting one extra pixel off the shrinked image when this happens?

I don't know if cutting an extra pixel off the shrunk image helps here. I thought this would be simpler, but maybe I'm overlooking something.

Thanks for the updates Kleis, I'm on the fence about whether to keep or remove fastShrinkOnLoad. Perhaps we should wait until libvips supports the proposed RGBX colourspace and therefore improve vips_reduce performance first?

Indeed, it would probably be better to wait for the RGBX improvement in libvips first. Using SIMD intrinsics in vips_reduce{h,v} will also speed that up.

Any thoughts/updates on getting this merged? I've been using @kleisauke 's branch and it's been working like a charm.

Thanks for testing. I'll remove the draft status once I've completed the TODO-items.

kleisauke · 2021-11-23T14:13:52Z

Although some new unit tests for animated images are still missing, I think this is now ready for reviewing. Note that this PR doesn't currently implement these shrink-on-load features of vips_thumbnail:

Page-based and ifd-based TIFF pyramids;
HEIC/AVIF thumbnails;
Page-based JP2K pyramids (requires vips 8.11);
Openslide levels.

Because that will require a lot of extra code paths (I think), which makes reviewing this PR difficult.

AlenVelocity · 2021-11-25T11:38:51Z

I wanted to test this PR by resizing GIFs and converting them to WebP.

The expected output was an animated WebP image which is contained in the image body.

Images used

Landscape
Potrait

Code

const image = sharp(filename, { animated: true })
await image.resize(512, 512, { 
    fit: 'contain',
    background: { r: 0, g: 0, b: 0, alpha: 0 } 
}).webp().toFile(join('results', filename.replace('gif', 'webp')))

Results

Sharp 0.29.4

Landscape	Potrait

github:kleisauke/sharp#vips-thumbnail

Landscape	Potrait

Is this a bug or am I doing something wrong?
Thanks for your time!

kleisauke · 2021-11-25T12:10:45Z

@AlenSaito1 Thanks for testing! Could you re-test with commit 3b79d8f?

The reason for this is that post-resize embedding/cropping modifies the image height, which breaks multi-page images. Commit 3b79d8f ensures that it will not alter the image height. A better solution would be to split the frames, and then perform this operation per-frame, but that comes at the expense of performance.

AlenVelocity · 2021-11-25T13:08:56Z

Thanks for the fast reply @kleisauke

I tried this again with the latest commit. But now, it works only if the image is a portrait.
If the image is a landscape, only the width gets set to the given value

Results

Landscape	Potrait

lovell

This is amazing Kleis, thank you, you've even done the multi-frame split and rejoin logic for the crop and embed paths! 🚀

lovell · 2021-11-25T19:58:45Z

src/pipeline.cc

          } else {
            // Reload WebP file
            image = VImage::webpload(const_cast<char*>(baton->input->file.data()), option);
          }
+        } else if (inputImageType == sharp::ImageType::SVG) {
+          option->set("unlimited", TRUE);


We should be able to use the value of the new baton->input->unlimited property here.

Great! Commit 70b1913 leverages this new property.

lovell · 2021-11-25T20:04:39Z

(I'm about to update the prebuilt binaries to include libspng 0.7.1 that will fix the failing ARM64 jobs.)

lovell · 2021-11-26T08:53:54Z

@kleisauke The main branch now uses the latest libvips 8.12.1 binaries so you should be good to rebase. Would you like to add any more unit tests before we merge this?

This should probably not be controlled by users, and should only change when the image is resized. This commit also changes the delay option to allow singular values (to change the delay for all frames).

This should be resolved by commit libvips/libvips@9c1003f and libvips/libvips@f92069b instead.

Inline with libvips' behaviour.

Saves a few ms.

kleisauke · 2021-11-26T09:36:31Z

I just rebased this PR, and will add some unit tests today. In addition that this PR doesn't port all shrink-on-load features, these post-resize operations are currently not supported on multi-page/animated images:

Smart-crop (is currently skipped for multi-page images);
Rotation (any abirtaty angle);
Affine transformation;
Compositing.

+ add unit tests for this.

Maybe this fixes the MacStadium CI error(?).

lovell · 2021-12-02T23:04:05Z

Thanks for the updates Kleis, this is ready to merge for v0.30.0 👍

My current thinking is that, if possible, it's worth aborting processing should the pipeline try to mix multi-page resizing with unsupported operations (smartcrop, rotation, affine, composite).

I'm happy add this logic, with unit tests, after we merge. Are there any other scenarios you can think of that should also be included?

kleisauke · 2021-12-03T12:44:50Z

Great! Currently smartcrop is silently ignored, see for example:

sharp/src/pipeline.cc

Line 464 in d72f4bc

} else if (nPages == 1) { // Skip smart crop for multi-page images

But it would indeed be better to abort processing for operations that doesn't support multi-page images. In addition to the operations mentioned above, I think pre-resize trimming is also unsupported on multi-page images.

There are also operations that can lead to surprising results. Blurring is one of them, see for example:
https://images.weserv.nl/static/banana.webp?n=-1&blur=10
(note the footprints at the top of the image)

lovell · 2021-12-10T20:33:38Z

Fantastisch, as always, thank you Kleis!

* Ports vips_thumbnail logic to sharp * Deprecates the pageHeight output option for WebP/GIF

kleisauke mentioned this pull request Jul 9, 2021

Implement scale option and automatic SVG scaling up #2787

Closed

lovell force-pushed the circle branch from d1b8462 to 3f6d9d6 Compare July 14, 2021 12:46

kleisauke force-pushed the vips-thumbnail branch from 82eaa8b to 1769bc8 Compare July 15, 2021 10:41

kleisauke changed the base branch from circle to master August 17, 2021 09:42

lovell mentioned this pull request Aug 26, 2021

Enhancement: improve animated/multipage image resizing #2275

Closed

kleisauke force-pushed the vips-thumbnail branch from 1769bc8 to 1de39a8 Compare August 27, 2021 10:35

kleisauke mentioned this pull request Aug 27, 2021

Ensure vips_thumbnail doesn't send an axis to 0 libvips/libvips#2416

Merged

lovell mentioned this pull request Sep 16, 2021

Encounter unexpect result when extend static gif and covert to webp #2890

Closed

Krislunde mentioned this pull request Oct 4, 2021

Gif support nicolashmln/strapi-plugin-responsive-image#14

Open

kleisauke mentioned this pull request Nov 1, 2021

Set page height only for multi-page images libvips/libvips#2506

Merged

kleisauke force-pushed the vips-thumbnail branch 2 times, most recently from 04a736d to b4ca8a4 Compare November 8, 2021 14:47

lovell mentioned this pull request Nov 18, 2021

Auto-detect animated sources unjs/ipx#53

Closed

kleisauke changed the base branch from master to main November 23, 2021 13:35

kleisauke force-pushed the vips-thumbnail branch from b4ca8a4 to e76b8d4 Compare November 23, 2021 13:52

kleisauke marked this pull request as ready for review November 23, 2021 14:13

kleisauke force-pushed the vips-thumbnail branch from 3b79d8f to 5dc150d Compare November 25, 2021 12:19

kleisauke mentioned this pull request Nov 25, 2021

GIF image embed extend not animating. libvips/php-vips#116

Closed

lovell reviewed Nov 25, 2021

View reviewed changes

kleisauke added 8 commits November 26, 2021 10:05

Port vips_thumbnail logic to sharp

6c58cde

Deprecate the pageHeight output option for WebP/GIF

f005dac

This should probably not be controlled by users, and should only change when the image is resized. This commit also changes the delay option to allow singular values (to change the delay for all frames).

Revert commit d359331

f4e964c

This should be resolved by commit libvips/libvips@9c1003f and libvips/libvips@f92069b instead.

Only set the pageHeight metadata property for multi-page images

e62576c

Inline with libvips' behaviour.

Support multi-page images when embedding/cropping

2304669

Add common fast paths for embedding

ffa8c5d

Saves a few ms.

Add common fast path for cropping

14286e0

Use new baton->input->unlimited property

70b1913

kleisauke force-pushed the vips-thumbnail branch from de64844 to 70b1913 Compare November 26, 2021 09:33

AlenVelocity mentioned this pull request Nov 26, 2021

Fix Full + Animated Stickers AlenVelocity/wa-sticker-formatter#29

Merged

kleisauke added 3 commits November 26, 2021 17:35

Add unit tests

79d6392

Support pre-resize extract

387365b

+ add unit tests for this.

Try to regenerate resize-crop-extract.jpg

d72f4bc

Maybe this fixes the MacStadium CI error(?).

lovell approved these changes Dec 10, 2021

View reviewed changes

lovell merged commit 513fb40 into lovell:main Dec 10, 2021

kleisauke deleted the vips-thumbnail branch December 11, 2021 11:01

lovell added this to the v0.30.0 milestone Dec 11, 2021

lovell mentioned this pull request Dec 11, 2021

Ensure operations without multi-page support reject rather than silently fail #3010

Merged

lovell added a commit that referenced this pull request Dec 11, 2021

Docs: changelog and refresh for #2789

8c0660d

Y2zz pushed a commit to Y2zz/sharp that referenced this pull request Feb 26, 2022

Improve multi-frame image resizing (lovell#2789)

42c0083

* Ports vips_thumbnail logic to sharp * Deprecates the pageHeight output option for WebP/GIF

Y2zz pushed a commit to Y2zz/sharp that referenced this pull request Feb 26, 2022

Docs: changelog and refresh for lovell#2789

c856fc3

martinj pushed a commit to aptoma/sharp that referenced this pull request Mar 31, 2022

Improve multi-frame image resizing (lovell#2789)

232b0ec

* Ports vips_thumbnail logic to sharp * Deprecates the pageHeight output option for WebP/GIF

martinj pushed a commit to aptoma/sharp that referenced this pull request Mar 31, 2022

Docs: changelog and refresh for lovell#2789

6846634

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port `vips_thumbnail` logic to sharp #2789

Port `vips_thumbnail` logic to sharp #2789

kleisauke commented Jul 9, 2021 •

edited

Loading

coveralls commented Jul 9, 2021 •

edited

Loading

ahukkanen commented Jul 9, 2021

lovell commented Jul 14, 2021

kleisauke commented Jul 15, 2021

ahukkanen commented Jul 15, 2021

lovell commented Jul 16, 2021

chrisdostert commented Aug 16, 2021 •

edited

Loading

kleisauke commented Aug 27, 2021

kleisauke commented Nov 23, 2021

AlenVelocity commented Nov 25, 2021

kleisauke commented Nov 25, 2021

AlenVelocity commented Nov 25, 2021

lovell left a comment

lovell Nov 25, 2021

kleisauke Nov 26, 2021

lovell commented Nov 25, 2021

lovell commented Nov 26, 2021

kleisauke commented Nov 26, 2021

lovell commented Dec 2, 2021

kleisauke commented Dec 3, 2021

lovell commented Dec 10, 2021

Port vips_thumbnail logic to sharp #2789

Port vips_thumbnail logic to sharp #2789

Conversation

kleisauke commented Jul 9, 2021 • edited Loading

coveralls commented Jul 9, 2021 • edited Loading

ahukkanen commented Jul 9, 2021

lovell commented Jul 14, 2021

kleisauke commented Jul 15, 2021

ahukkanen commented Jul 15, 2021

lovell commented Jul 16, 2021

chrisdostert commented Aug 16, 2021 • edited Loading

kleisauke commented Aug 27, 2021

kleisauke commented Nov 23, 2021

AlenVelocity commented Nov 25, 2021

Images used

Code

Results

kleisauke commented Nov 25, 2021

AlenVelocity commented Nov 25, 2021

lovell left a comment

Choose a reason for hiding this comment

lovell Nov 25, 2021

Choose a reason for hiding this comment

kleisauke Nov 26, 2021

Choose a reason for hiding this comment

lovell commented Nov 25, 2021

lovell commented Nov 26, 2021

kleisauke commented Nov 26, 2021

lovell commented Dec 2, 2021

kleisauke commented Dec 3, 2021

lovell commented Dec 10, 2021

Port `vips_thumbnail` logic to sharp #2789

Port `vips_thumbnail` logic to sharp #2789

kleisauke commented Jul 9, 2021 •

edited

Loading

coveralls commented Jul 9, 2021 •

edited

Loading

chrisdostert commented Aug 16, 2021 •

edited

Loading