Add smart animated format conversion. #2588

JimBobSquarePants · 2023-11-24T12:43:47Z

Prerequisites

I have written a descriptive pull-request title
I have verified that there are no overlapping pull-requests open
I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
I have provided test coverage for my change (where applicable)

Description

Note: Most of the file changes here are new references images that were generated to cater for changes in the color distance calculator that improved lookup accuracy.

This PR adds some internal tooling that allows us to convert between animated image formats without the user having to specify the mechanics for each frame to ensure converting between formats returns an equivalent result.

For v4 I plan to update the metadata formats to introduce public bridging types and methods to create smart defaults when switching formats but for now I had to use an internal, non-extensible approach.

For v3.1 we will support bi-directional conversion between gif, png, and webp formats. Full color table mapping is performed between formats when present as well as duration, and disposal, blend modes where applicable.

Here's an example of an automatic conversion.

Gif (45.2KB)

Png (33.3KB)

In addition to the conversion functionality all 3 animated formats have been enhanced with shared methodology that allows the optimization of the frames to only include changed data. This prevents the potential explosion in file size (multi MBs) that can be caused when re-encoding the frames that are coalesced during decode.

This example from gifiddle demonstrates the active encoded area of a frame following that optimization.

For many gifs this actually leads to an improvement over the incoming file size. For example, the original of the gif above is 52.3KB

JimBobSquarePants · 2023-11-25T08:39:41Z

@antonfirsov @tocsoft I need your help if you have time?

I'm seeing allocation cleanup failure reports on Linux/Mac and I can recreate them on my old MacBook. Thing is... I've touched nothing related to memory management nor does this test touch code I've changed with my PR.

Is there any way to easily diagnose the cause? I tried using dotmemory Unit in Rider but it won't give the option to run the test in the manner that allows profiling memory.

Running this test in isolation is enough to cause failure.
[xUnit.net 00:00:27.79] TiffDecoder_CanDecode_JpegCompressed(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32]) [FAIL]

EDIT.... OK. It was failing locally but now it suddenly isn't. Could it be a memory pressure thing triggered by my additional tests?

EDIT... It was ME!!

src/ImageSharp/Formats/AnimationUtilities.cs

src/ImageSharp/Processing/Processors/Quantization/EuclideanPixelMap{TPixel}.cs

stefannikolei · 2023-11-25T10:54:23Z

Forget this...

I removed my local copy of the repo and did a clean checkout. Now everything works on the main branch.. strange

@antonfirsov @tocsoft I need your help if you have time?

I'm seeing allocation cleanup failure reports on Linux/Mac and I can recreate them on my old MacBook. Thing is... I've touched nothing related to memory management nor does this test touch code I've changed with my PR.

Is there any way to easily diagnose the cause? I tried using dotmemory Unit in Rider but it won't give the option to run the test in the manner that allows profiling memory.

Running this test in isolation is enough to cause failure. [xUnit.net 00:00:27.79] TiffDecoder_CanDecode_JpegCompressed(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32]) [FAIL]

EDIT.... OK. It was failing locally but now it suddenly isn't. Could it be a memory pressure thing triggered by my additional tests?

I tried running main first on my Mac. But not even there all tests go green.

     Encode_AnimatedLossy<Rgba32>(provider: Webp/leo_animated_lossy.webp[Rgba32])

SixLabors.ImageSharp.ImageFormatException
Read unexpected webp chunk data
at SixLabors.ImageSharp.Formats.Webp.WebpThrowHelper.ThrowImageFormatException(String errorMessage) in /Users/stefannikolei/projects/github/sixlabors/ImageSharp/src/ImageSharp/Formats/Webp/WebpThrowHelper.cs:line 14

   CanLimitHwIntrinsicFeaturesWithIntrinsicsParam
   CanLimitHwIntrinsicFeaturesWithSerializableAndIntrinsicsParams
   CanLimitHwIntrinsicFeaturesWithSerializableParam

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException
Remote process failed with an unhandled exception.
Child exception:
Xunit.Sdk.FalseException: Assert.False() Failure
Expected: False
Actual: True

stefannikolei · 2023-11-25T14:56:41Z

following tests are failing on my machine

     GifDecoderTests
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/cheers.gif[Rgba32], expectedFrameCount: 93)
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/issues/issue403_baddescriptorwidth.gif[Rgba32], expectedFrameCount: 36)
     Issue1668_InvalidColorIndex<Rgba32>(provider: Gif/issues/issue1668_invalidcolorindex.gif[Rgba32])
   PngEncoderTests
     Encode_APng<Rgba32>(provider: Png/animated/apng.png[Rgba32])     
    TiffDecoderTests
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_deflate_withPreview.tiff[Rgba32])
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_lzw.tiff[Rgba32])
     TiffDecoder_CanDecode_JpegCompressed<Rgba32>(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32])

I tried debugging it.

In my eyes (looking at the failing png test) The problem starts to manifest here:

In all other iterations the width was 32 and resulted in bytesPerFrameScanLine = 129. Here the width is 10

A naive fix (the unit test passes after this:

JimBobSquarePants · 2023-11-26T12:16:25Z

following tests are failing on my machine

     GifDecoderTests
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/cheers.gif[Rgba32], expectedFrameCount: 93)
     Decode_VerifyRootFrameAndFrameCount<Rgba32>(provider: Gif/issues/issue403_baddescriptorwidth.gif[Rgba32], expectedFrameCount: 36)
     Issue1668_InvalidColorIndex<Rgba32>(provider: Gif/issues/issue1668_invalidcolorindex.gif[Rgba32])
   PngEncoderTests
     Encode_APng<Rgba32>(provider: Png/animated/apng.png[Rgba32])     
    TiffDecoderTests
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_deflate_withPreview.tiff[Rgba32])
     DecodeMultiframe<Rgba32>(provider: Tiff/multipage_lzw.tiff[Rgba32])
     TiffDecoder_CanDecode_JpegCompressed<Rgba32>(provider: Tiff/twain-rgb-jpeg-with-bogus-ycbcr-subsampling.tiff[Rgba32])

I tried debugging it.

In my eyes (looking at the failing png test) The problem starts to manifest here:

In all other iterations the width was 32 and resulted in bytesPerFrameScanLine = 129. Here the width is 10

A naive fix (the unit test passes after this:

Well spotted! Yes, that's definitely a bug. The interlaced version of the method was doing the correct thing.

gfoidl

Code-wise LGTM.

To the implementation itself I can't say too much as I'm missing knowledge here.

gfoidl · 2023-11-29T09:28:24Z

src/ImageSharp/Processing/Processors/Quantization/EuclideanPixelMap{TPixel}.cs

    /// </para>
    /// </remarks>
    private unsafe struct ColorDistanceCache : IDisposable
    {
        private const int IndexRBits = 5;
        private const int IndexGBits = 5;
-        private const int IndexBBits = 6;
+        private const int IndexBBits = 5;


Is there a test that unveils that "bug" (was this a bug?)?

I was something I just caught doing a visual test which is hard to test for.

The cache was only filled to a maximum value of ~96% which led to matching where it shouldn't. This led to transparent areas in the frame following deduplication. Originally, I was going to simply add more precision for a single component (blue) which fixed the issue, but I discovered that with the new index determinaiton code the cache fills to a maximum of ~98% and the problem simply went away.

JimBobSquarePants added 8 commits November 15, 2023 12:40

Wire up connectors and gif encoder

328e046

Complete Webp and add tests

a486558

Default loop count should be 1

7f4b457

Merge branch 'main' into js/animation-synergy

aea44fa

Deduper works

958c9c9

Enable dedup for png

a42f6b6

Add dedup to webp

cc0727b

Update tests, fix issues

4b852e6

JimBobSquarePants added enhancement area:performance formats:gif labels Nov 24, 2023

JimBobSquarePants added this to the v3.1.0 milestone Nov 24, 2023

JimBobSquarePants requested review from antonfirsov, brianpopow, tocsoft and dlemstra November 24, 2023 12:44

JimBobSquarePants added 3 commits November 24, 2023 23:11

Tweak bounds clamping

4f8ea7f

Merge branch 'main' into js/animation-synergy

78b1cad

Use correct buffer dimensions

c306c56

gfoidl reviewed Nov 25, 2023

View reviewed changes

Fix scanline lengths

6cda7b0

JimBobSquarePants added 4 commits November 26, 2023 22:40

Update PngDecoderCore.cs

90fa817

Remove duplicate condition check

4029b15

Remove "new"

44b0311

Try disabling new high memory tests on failing platforms.

55e69c7

JimBobSquarePants added 9 commits November 28, 2023 22:39

Optimize and fix deduper

a6f96f7

Reduce memory pressure on gif decoder tests

6bfabe9

More memory pressure reductions

9677be0

More memory pressure reduction

35d55b5

Reduce memory usage in pixel map

e35e9a8

Enable Sse2, simplify

5da17f3

Update AnimationUtilities.cs

10cd3d5

Disable Sse2 for now.

f4dc0fc

Fix Sse2 offset

43c8614

JimBobSquarePants requested a review from gfoidl November 29, 2023 03:37

JimBobSquarePants added 3 commits November 29, 2023 16:50

Add note

cf4106b

Merge branch 'main' into js/animation-synergy

3424efa

Merge branch 'main' into js/animation-synergy

3b759a5

gfoidl approved these changes Nov 29, 2023

View reviewed changes

JimBobSquarePants merged commit 60e0298 into main Nov 29, 2023
8 checks passed

JimBobSquarePants deleted the js/animation-synergy branch November 29, 2023 11:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add smart animated format conversion. #2588

Add smart animated format conversion. #2588

JimBobSquarePants commented Nov 24, 2023

JimBobSquarePants commented Nov 25, 2023 •

edited

stefannikolei commented Nov 25, 2023 •

edited

stefannikolei commented Nov 25, 2023 •

edited

JimBobSquarePants commented Nov 26, 2023

gfoidl left a comment

gfoidl Nov 29, 2023

JimBobSquarePants Nov 29, 2023

Add smart animated format conversion. #2588

Add smart animated format conversion. #2588

Conversation

JimBobSquarePants commented Nov 24, 2023

Prerequisites

Description

JimBobSquarePants commented Nov 25, 2023 • edited

stefannikolei commented Nov 25, 2023 • edited

stefannikolei commented Nov 25, 2023 • edited

JimBobSquarePants commented Nov 26, 2023

gfoidl left a comment

Choose a reason for hiding this comment

gfoidl Nov 29, 2023

Choose a reason for hiding this comment

JimBobSquarePants Nov 29, 2023

Choose a reason for hiding this comment

JimBobSquarePants commented Nov 25, 2023 •

edited

stefannikolei commented Nov 25, 2023 •

edited

stefannikolei commented Nov 25, 2023 •

edited