SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.Encode: Experiencing High Native Memory Allocation in Release Mode (and not Debug)

[Mike-E-angelo]

I am experiencing an issue over the past few days that I would like some guidance in diagnosing before I report it.

To explain the situation, I am running a data migration tool that is converting approximately 120GB of image data from v1 to v2 of our product. I am running Parallel.ForEachAsync to process each row of data, and have 32 processors + 64GB of RAM on my machine.

In Debug mode, this seems to behave as expected: managed memory is around ~350-450MB and the total memory hangs at about 1GB maximum, dropping as low as 550MB.

I can confirm that SixLabors.ImageSharp.Diagnostics.MemoryDiagnostics.TotalUndisposedAllocationCount is zero for each pass.

When I run the application in Release mode, however, I see wildly different results. Managed memory again stays at around ~350-450MB but the total memory utilization will continually increase and explode up to 26GB after about 10 minutes of runtime.

I have been using ReSharper/dotTrace to help out, and it seems this is happening in the following area:

  100%   All Calls  •  41,596 MB
    64.9%   MoveNext  •  27,001 MB  •  DragonSpark.Azure.Storage.Write+<Get>d__2.MoveNext()
      64.0%   SaveWebpProfile+<>c__DisplayClass0_0.<.ctor>b__0  •  26,619 MB  •  Starbeam.Applications.Production.Files.Storage.Images.SaveWebpProfile+<>c__DisplayClass0_0.<.ctor>b__0(Image, Stream, CancellationToken)
        64.0%   MoveNext  •  26,619 MB  •  Starbeam.Applications.Production.Files.Storage.Images.Copy+<Get>d__4.MoveNext()
          63.4%   EncodeAsync  •  26,356 MB  •  SixLabors.ImageSharp.Formats.ImageEncoder.EncodeAsync(Image, Stream, CancellationToken)
            63.4%   MoveNext  •  26,356 MB  •  SixLabors.ImageSharp.Formats.ImageEncoder+<EncodeWithSeekableStreamAsync>d__8`1.MoveNext()
              63.4%   <EncodeWithSeekableStreamAsync>g__DoEncodeAsync|8_0  •  26,356 MB  •  SixLabors.ImageSharp.Formats.ImageEncoder.<EncodeWithSeekableStreamAsync>g__DoEncodeAsync|8_0(Stream, ref <>c__DisplayClass8_0)
                63.4%   Encode  •  26,356 MB  •  SixLabors.ImageSharp.Formats.Webp.WebpEncoderCore.Encode(Image, Stream, CancellationToken)
                  62.7%   Encode  •  26,091 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.Encode(Stream, ImageFrame, Rectangle, WebpFrameMetadata, bool, Image)
                  ► 26.8%   StatLoop  •  11,166 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.StatLoop(int, int, int, int)
                  ► 26.3%   Decimate  •  10,932 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.Decimate(Vp8EncIterator, ref Vp8ModeScore, Vp8RdLevel)
                  ► 4.47%   EncodeAlpha  •  1,858 MB  •  SixLabors.ImageSharp.Formats.Webp.AlphaEncoder.EncodeAlpha(Buffer2DRegion, Configuration, MemoryAllocator, bool, bool, ref int)
                  ► 4.02%   CodeResiduals  •  1,670 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.CodeResiduals(Vp8EncIterator, Vp8ModeScore, Vp8Residual)
                  ► 0.42%   WriteEncodedImageToStream  •  173 MB  •  SixLabors.ImageSharp.Formats.Webp.BitWriter.Vp8BitWriter.WriteEncodedImageToStream(Stream)
                  ► 0.36%   MbAnalyze  •  151 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.MbAnalyze(Vp8EncIterator, Span, ref int)
                  ► 0.24%   EncodeAlpha  •  101 MB  •  SixLabors.ImageSharp.Formats.Webp.AlphaEncoder.EncodeAlpha(Buffer2DRegion, Configuration, MemoryAllocator, bool, bool, ref int)
                    0.07%   coreclr.dll  •  29 MB
                  ► 0.01%   Vp8EncIterator..ctor  •  5.5 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8EncIterator..ctor(Vp8Encoder)
                    <0.01%   Write  •  2.5 MB  •  System.IO.MemoryStream.Write(ReadOnlySpan)
                  ► <0.01%   Vp8ModeScore..ctor  •  1.4 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8ModeScore..ctor()
                  ► <0.01%   ConvertRgbToYuv  •  1.2 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.YuvConversion.ConvertRgbToYuv(Buffer2DRegion, Configuration, MemoryAllocator, Span, Span, Span)
                  ► <0.01%   EncodeHeader  •  0.5 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8Encoder.EncodeHeader(Image, Stream, bool, bool)
                  ► <0.01%   EndWriteChunk  •  0.4 MB  •  SixLabors.ImageSharp.Formats.Webp.RiffHelper.EndWriteChunk(Stream, long)
                  0.29%   coreclr.dll  •  121 MB
                ► 0.25%   Vp8EncProba..ctor  •  105 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossy.Vp8EncProba..ctor()
                ► 0.08%   Allocate  •  32 MB  •  SixLabors.ImageSharp.Memory.MemoryAllocator.Allocate(int, AllocationOptions)
                ► 0.01%   Encode  •  5.8 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossless.Vp8LEncoder.Encode(ImageFrame, Rectangle, WebpFrameMetadata, Stream, bool)
                ► <0.01%   Vp8LEncoder..ctor  •  0.7 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossless.Vp8LEncoder..ctor(MemoryAllocator, Configuration, int, int, uint, bool, WebpEncodingMethod, TransparentColorMode, bool, int)
                ► <0.01%   EncodeFooter  •  0.3 MB  •  SixLabors.ImageSharp.Formats.Webp.Lossless.Vp8LEncoder.EncodeFooter(Image, ref WebpVp8X, bool, Stream, long)
                  <0.01%   Initialize  •  0.1 MB  •  System.Collections.Generic.Dictionary`2.Initialize(int)
              ► <0.01%   Encode  •  0.2 MB  •  SixLabors.ImageSharp.Formats.Webp.WebpEncoder.Encode(Image, Stream, CancellationToken)
        ► 0.37%   AcceptAsync  •  152 MB  •  SixLabors.ImageSharp.Image`1.AcceptAsync(IImageVisitorAsync, CancellationToken)
        ► 0.20%   AcceptAsync  •  82 MB  •  SixLabors.ImageSharp.Image`1.AcceptAsync(IImageVisitorAsync, CancellationToken)
          0.04%   CopyToAsync  •  15 MB  •  System.IO.MemoryStream.CopyToAsync(Stream, int, CancellationToken)
          0.03%   MoveNext  •  13 MB  •  Azure.Storage.Shared.StorageWriteStream+<FlushAsync>d__33.MoveNext()
          <0.01%   GetStateMachineBox  •  0.1 MB  •  System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.GetStateMachineBox(ref !!0, ref Task)
          <0.01%   coreclr.dll  •  0.1 MB

Again it seems that I am disposing of all Images as TotalUndisposedAllocationCount suggests. I am wondering if there's another consideration here or if this may be a bug that needs further diagnosing and reporting.

I am not too familiar with this (unmanaged) area so if I can get some guidance on how to proceed, I would greatly appreciate it. I will do my best to get you the information required. Thank you for any assistance in helping to track down this issue.
`

[JimBobSquarePants]

Thanks for the detailed report. The trace does not show a native memory leak, and it does not show ImageSharp holding 26 GB. Here is why.

The numbers in the call tree are cumulative. dotTrace sums every allocation made while profiling. "All Calls 41,596 MB" is the total allocated over the whole run, not the memory in use. That number cannot be compared with a live process-memory reading.

The tree shows managed allocations, not native ones. Frames such as GetStateMachineBox, Dictionary.Initialize, and MemoryStream.Write only allocate managed objects. The lossy encoder creates small score and residual objects for every macroblock. A large image has tens of thousands of macroblocks, so this adds up to gigabytes over thousands of images. The GC collects these objects, which is why your managed heap stays flat at 350 to 450 MB. I am removing that churn in a separate change, but it is not a leak.

TotalUndisposedAllocationCount at zero means your code disposed every image. It does not count memory that the pool retains. By default the pool keeps up to 1/8 of physical RAM, which is 8 GB on your machine. Pooled memory is trimmed over time, and it is capped, so the pool alone cannot reach 26 GB.

Debug and Release run the same ImageSharp binary. Only your own code, the JIT, and throughput change. A Debug run under the debugger is much slower. Ten minutes of Debug processes far fewer images than ten minutes of Release. Please compare both modes after the same number of images, not after the same time.

To find the actual source of the growth, I need the following:

1. The ImageSharp version, the .NET version, and the GC mode (Server or Workstation).
2. How you measure "managed memory" and "total memory". Please log GC.GetGCMemoryInfo().TotalCommittedBytes and Environment.WorkingSet at the same moments. If committed GC bytes track the total, the growth is GC heap that is committed but not live. Profilers report that space as unmanaged.
3. The typical image dimensions, and the MaxDegreeOfParallelism you use.
4. Any MemoryAllocator configuration you set.
5. The result of two experiments:
   • Set Configuration.Default.MemoryAllocator = MemoryAllocator.Create(new MemoryAllocatorOptions { MaximumPoolSizeMegabytes = 256 }); before the migration starts. If the total drops, the pool was the retained memory.
   • Run with the environment variable DOTNET_GCConserveMemory=5. If the total drops, the GC was the retained memory.

If neither experiment changes the total, the memory is outside ImageSharp. In that case, please check the MemoryStream buffers that hold each encoded image before the Azure upload.

[Mike-E-angelo]

Hi @JimBobSquarePants thank you for the really quick reply! I really appreciate it, along with the detailed analysis of Debug vs. Release. I agree with your explanation.

You're also 100% correct about dotTrace's reporting. I may have gotten a little bedazzled by its analysis, since this isn't a scenario I've had to dive into much. But you are correct; its reporting is managed and has nothing to do with unmanaged (though from the outset I was thinking the managed was somehow contributing to the unmanaged).

However, all that said, I am happy to report that after several days of hunting around, I finally found the reason for my grief, and it has nothing to do with ImageSharp:

https://github.com/DragonSpark/Framework/blob/06b8f3de81f70957758733a2327cbbdeec132782/DragonSpark.Drawing/LoadImage.cs#L27

I forgot to dispose of a copied Bitmap. 🤦 After disposing of that, my memory utilization jumps up to maybe 900MB but then back down to 400-500MB, exactly as expected:

Using ReSharper's Monitoring window, btw. 🤓👍

Thank you again for your great support and patience as I waded through unfamiliar territory. I greatly appreciate it! 🙏

[JimBobSquarePants]

Great find, and thanks for coming back to close the loop!

That explains the symptoms well. A System.Drawing Bitmap holds its pixels in native GDI memory, so an undisposed copy grows exactly the "unmanaged" number you were watching, while the managed heap stays flat. Glad the graph looks healthy now.

The allocation churn the trace showed is real, even though it was not your problem. I have a change ready that removes the per-macroblock allocations in the lossy encoder, so that path will allocate a lot less in the next release. Your profile was a useful nudge, so thank you for that too.

Happy migrating!