WebP compression artefacts relating to premultiplication rounding

[violetviolinist]

Feature request

What are you trying to achieve?

The recent change related to (un)premultiplication in resize() is producing artefacts in one of our images (messing up color in some pixels) when resize() + extend() is applied on it. It is a 158 x 31 image; a word of text on a fully transparent background, all letters having the same darg-gray (#333333) color. I verified this change to be the cause by reverting it and rebuilding sharp, and applying the resize again.

We would love to have an option added to the resize() operation to go back to not using integer (un)premultiplication. I would be happy to contribute with a PR. Just wanted to collect thoughts first.

When you searched for similar feature requests, what did you find that might be related?

NA

What would you expect the API to look like?

A new field useIntegerPremultiply added to resize(). Will default to true :

resize(100, 100, {
  useIntegerPremultiply: false
})

[lovell]

Please can you provide a complete, standalone code sample that allows someone else to reproduce the problem you are experiencing as I'd like to understand this first before jumping into a solution.

[violetviolinist]

Here's sample code and the input image to reproduce the issue. The two outputs to compare are from v0.32.1 and v0.31.3. Although I have singled out the above mentioned commit to be the culprit by recompiling v0.32.1 with only that change reverted in pipeline.cc.

Note: You may have to zoom in a little into the outputs to see the distorted pixels.

  (async () => {
    const origBuffer = fs.readFileSync(`./textsample.png`)

    const outBuffer = await sharp(origBuffer)
    .resize(800, 500, {
      fit: 'inside',
    })
    .extend({
      background: {r: 255, g: 255, b: 255, alpha: 0},
      bottom: 121,
      left: 0,
      right: 0,
      top: 120,
    })
    .webp()
    .toBuffer()
  
    fs.writeFileSync("./out.webp", outBuffer)
  })()

Input image: textsample.png

[lovell]

These look like WebP compression artefacts. For WebP output images that contain text, I recommend using either lossless or nearLossless.

[violetviolinist]

Yes, the artefacts seem to be limited to WebP encoding only. But even for WebP encoding, they did appear only after the premultiplication change. As for conditionally using lossless or nearLossless for images with text, we cannot do that since we do not always know the content of the image being processed. And we cannot afford the higher processing times if we were to use nearLossless on all images unconditionally.

Which is why making integer premultiplication configurable seems to be the only solution for us.

[lovell]

Does increasing the quality help? Does using smartSubsample help? These settings may be relevant if some images will contain text.

https://sharp.pixelplumbing.com/api-output#webp

If not, this might provide a good test image (as a PNG, without the artefacts) to use to ask the libwepb maintainers upstream.

[violetviolinist]

Does increasing the quality help? Does using smartSubsample help? These settings may be relevant if some images will contain text.

Yes. Increasing quality and using nearLossless helps. However, one popular use case of Sharp (including ours) is to resize/optimize images in real-time. Detecting the presence of text in an image is not computationally cheap and desirable in the processing pipeline.

Please confirm if my understanding is correct - With everything else being the same, we get the better output without premultiplication rounding vs with premultiplication rounding. That means the upstream library is producing consistent results.

If the above is true, wouldn't it be better to make premultiplication rounding configurable behind an option with the default value set to true to match the current behaviour of the library that works well for most of the images?

[lovell]

If libwebp is introducing compression artefacts then that feels like the right place to address this if possible. What did the libwebp maintainers say when you asked them?

[violetviolinist]

I haven't asked them because without sharp's premultiplication rounding, libwebp is producing the artefact-free output. And since the rounding was introduced in sharp, I thought this would be the right place to ask.

Also, was this rounding change expected to be completely harmless across different combinations w/ resize()? I am trying to understand the reasoning behind incorporating this change across the board without the option to go back. At this point, taking this up with libwebp maintainers may or may not solve the issue. But allowing to opt-out of the rounding within sharp will definitely fix it.

[lovell]

The output of your sample code, saved as a lossless PNG, does not appear to have the compression artefacts:

Using this PNG without artefacts as input via the libwebp command line tools (and not sharp):

$ cwebp -version
1.2.4

$ cwebp in.png -q 80 -o out.webp
Saving file 'out.webp'
File:      in.png
Dimension: 800 x 377 (with alpha)
Output:    10364 bytes Y-U-V-All-PSNR 52.18 99.00 99.00   53.95 dB
           (0.27 bpp)
block count:  intra4:        103  (8.58%)
              intra16:      1097  (91.42%)
              skipped:      1036  (86.33%)
bytes used:  header:             98  (0.9%)
             mode-partition:    913  (8.8%)
             transparency:     8021 (99.0 dB)
 Residuals bytes  |segment 1|segment 2|segment 3|segment 4|  total
    macroblocks:  |       3%|      11%|       3%|      84%|    1200
      quantizer:  |      27 |      27 |      26 |      17 |
   filter level:  |       8 |      63 |       5 |       2 |
Lossless-alpha compressed size: 8020 bytes
  * Header size: 155 bytes, image data size: 7865
  * Precision Bits: histogram=5 transform=5 cache=0
  * Palette size:   256

$ dwebp out.webp -o out.png
Decoded out.webp. Dimensions: 800 x 377  (with alpha). Format: lossy. Now saving...
Saved file out.png

...produces an output image with compression artefacts:

The intermediate WebP image is attached here in a zip file so GitHub doesn't mangle it.

out.webp.zip

A small change to rounding behaviour in sharp has allowed the creation of a PNG image that can then be used to expose these compression artefacts. It is likely that other input images, those that have not been processed via sharp, will also expose this behaviour of libwebp.

If inter-pipeline precision within sharp is of concern, I recommend you take a look at pipelineColourspace.

[lovell]

I hope this information helped. Please feel free to re-open with more details if further assistance is required.