How do I use ImageSharp with large orthomosaics?

[aevitas]

Hi,

I've been a long time user of ImageSharp for all my imaging needs, so when I started a recent project using satellite imagery and drone footage, I once again came back to this library.

The images I'm working with are large geographical TIFF images, also known as orthomosaics. They can be stitched together from numerous other images, and as a result can become quite large (some of our current images are north of 1.5GB).

I'd like to use ImageSharp to transform, transcode and "tile" these images into smaller images, but when I attempt to load one of these images, I get the following exception:

Unhandled exception. SixLabors.ImageSharp.InvalidImageContentException: Cannot decode image. Failed to allocate buffers for possibly degenerate dimensions: 0x0.
 ---> SixLabors.ImageSharp.Memory.InvalidMemoryOperationException: Attempted to allocate a buffer of negative length=-227135488.
   at SixLabors.ImageSharp.Memory.InvalidMemoryOperationException.ThrowNegativeAllocationException(Int64 length)
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator.Allocate[T](Int32 length, AllocationOptions options)

Debugging this, I can see the following number of bytes being allocated every time:

1. 62424
2. 15606
3. 15606
4. 262144
5. -227135488

Which I assume is four metadata allocations, followed by an allocation for the actual (overflowed size) image data.

Given that the TIFF I'm trying to load is 740MB, I fail to see why it would need to allocate over 2.14GB of memory to process the image. Other libraries such as GDAL are perfectly fine loading these images in an instant, but don't offer the image manipulation I need.

Is this a limitation with ImageSharp specifically? Is there something I'm doing wrong and is there a way to use these images with ImageSharp?

I've explored making my own fork of ImageSharp and replacing the allocation with just flat direct allocations as I'd be running this from a command line so the heap would never be long-lived, but it seemed like a non-trivial modification so I'd like to make sure there aren't any more sensible approaches.

Can provide sample files if needed.

[JimBobSquarePants]

Hi @aevitas

Are you using the latest version v3.1.6? That negative number looks to me like we are suffering from some sort of arithmetic overflow issue. Were there additional lines in the stack trace?

[aevitas]

Yes, I'm using ImageSharp 3.1.6. I think it's overflowing somewhere in DecodeTilesChunky<SixLabors.ImageSharp.PixelFormats.Rgba32>:

UniformUnmanagedMemoryPoolMemoryAllocator.Allocate<byte>() UniformUnmanagedMemoryPoolMemoryAllocator.cs:line 86
TiffDecoderCore.DecodeTilesChunky<SixLabors.ImageSharp.PixelFormats.Rgba32>() at /_/src/ImageSharp/Formats/Tiff/TiffDecoderCore.cs:line 642
TiffDecoderCore.DecodeImageWithTiles<SixLabors.ImageSharp.PixelFormats.Rgba32>()
TiffDecoderCore.DecodeFrame<SixLabors.ImageSharp.PixelFormats.Rgba32>()
TiffDecoderCore.Decode<SixLabors.ImageSharp.PixelFormats.Rgba32>()
ImageDecoderCore.Decode<SixLabors.ImageSharp.PixelFormats.Rgba32>()
TiffDecoder.Decode<SixLabors.ImageSharp.PixelFormats.Rgba32>()
TiffDecoder.Decode()
ImageDecoder.<>c__DisplayClass1_0.<Decode>b__0()
ImageDecoder.<WithSeekableStream>g__PerformActionAndResetPosition|11_0<SixLabors.ImageSharp.Image>()
ImageDecoder.WithSeekableStream<SixLabors.ImageSharp.Image>()
ImageDecoder.Decode()
Image.Decode()
Image.<>c__DisplayClass80_0.<Load>b__0()
Image.WithSeekableStream<SixLabors.ImageSharp.Image>()
Image.Load()
Image.Load()
Image.Load()
Program.<Main>$()

If needed I can provide a fairly minimal application to reproduce with some of the actual TIFF data I'm using.

[antonfirsov]

That would help us a lot!

[aevitas]

That would help us a lot!

This code will throw the exception on the Image.Load call:
Archive.zip

When used with this orthomosaic

Let me know if you need me to convert this discussion into an issue.

[aevitas]

I've been diving a little deeper into this, and I've found that uncompressedPixelBuffer in DecodeTilesChunky overflows. It requires 4067831808 bytes to allocate the full image, which seems impractical.

I'm not that well-versed in the nitty gritty of the TIF file format, but would it perhaps be possible to determine tiles using specific regions of the image so that I'd never have to allocate the full image data, but only specific sections?

Is it even possible to load images this large since considering the maximum length of an Array is int.MaxValue?

[JimBobSquarePants]

Yep. It's possible to remove that allocation completely and decode each tile row directly to the image.

[aevitas]

Yep. It's possible to remove that allocation completely and decode each tile row directly to the image.

Is there an example of another decoder that follows this approach? I'm getting some strange results with the TIFF format across the board. For example, here's a TIFF file I saved as a JPEG using Photoshop and ImageSharp respectively:

Photoshop:

ImageSharp:

Doesn't seem quite right either. JPEG in, JPEG out with ImageSharp works fine for this image, but TIFF -> JPEG or any other format is just scan lines.

If there's a reference approach and it doesn't require too much specialised knowledge, maybe I can take a look and see if I can streamline the implementation.

[JimBobSquarePants]

Was that test made with my changes? If so that indicates a bug in my offsetting.

If it’s v3.1.6 then the bug lies elsewhere. The best reference source is libtiff.

[aevitas]

Was that test made with my changes? If so that indicates a bug in my offsetting.

If it’s v3.1.6 then the bug lies elsewhere. The best reference source is libtiff.

The artifacts appear when using 3.1.6. I'll run it again with the changes in js/chunk-tiff-allocation tomorrow.

[JimBobSquarePants]

Ah ok, possibly different TIFF type. Please open issues for any TIFF files we cannot open.

[aevitas]

With the changes proposed in #2874 , these images now load properly. However, when loading the larger ones, I run into the allocation limit set by MemoryGroupAllocationLimitBytes as 5545421976 bytes are being allocated for an image with dimensions 46941x29534 which exceeds the 4GB limit. When I increase this limit to 8GB, the image loads properly.

Would it be problematic if I'd run ImageSharp with such a high maximum allocation size?

[JimBobSquarePants]

Not if your machine can handle it. I'll continue to investigate optimization opportunities but those are massive images.

[mfeemster]

I am trying to decode this extremely large 14GB TIF file: https://rubinobservatory.org/gallery/collections/first-look-gallery/mlis3sriah6pn6nfr5ecp46h3i

and get the same error. My machine has 64GB RAM and I set the buffer sizes to 40,000 (40GB) and it still gives me the "Failed to allocate buffers for possibly degenerate dimensions: 0x0." error.

My aim is to split the image into smaller tiles, but it appears that I must load the original image fully into memory before doing the split operations.

So is there any way to fix the large TIF problem? Or is there a way to only load a piece of the image at a time?

Thanks.

[JimBobSquarePants]

Can you please run the image against the source here and provide a stack trace? The fact that it says 0x0 suggests a separate issue.

[mfeemster]

I pulled the latest code, built and ran this code. Note that I set the memory limit to 20GB, which ought to be enough for this 14GB image.

try
{
	Configuration.Default.MemoryAllocator = MemoryAllocator.Create(new MemoryAllocatorOptions()
	{
		MaximumPoolSizeMegabytes = 20000,
		AllocationLimitMegabytes = 20000,
	});

	var filename = @"D:\Temp Downloads\Vera Rubin Cosmic Treasure Chest Im1.tif";
	var image = SixLabors.ImageSharp.Image.Load<Rgba32>(filename);
}
catch (Exception ex)
{
	Console.WriteLine(ex);
}

It threw an exception with this stack trace:

SixLabors.ImageSharp.InvalidImageContentException: Cannot decode image. Failed to allocate buffers for possibly degenerate dimensions: 0x0.
 ---> SixLabors.ImageSharp.Memory.InvalidMemoryOperationException: Attempted to allocate a buffer of negative length=-2037097840.
   at SixLabors.ImageSharp.Memory.InvalidMemoryOperationException.ThrowNegativeAllocationException(Int64 length) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Memory\InvalidMemoryOperationException.cs:line 32
   at SixLabors.ImageSharp.Memory.UniformUnmanagedMemoryPoolMemoryAllocator.Allocate[T](Int32 length, AllocationOptions options) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Memory\Allocators\UniformUnmanagedMemoryPoolMemoryAllocator.cs:line 88
   at SixLabors.ImageSharp.Formats.Tiff.TiffDecoderCore.DecodeStripsChunky[TPixel](ImageFrame`1 frame, Int32 width, Int32 height, Int32 rowsPerStrip, Span`1 stripOffsets, Span`1 stripByteCounts, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\Tiff\TiffDecoderCore.cs:line 513
   at SixLabors.ImageSharp.Formats.Tiff.TiffDecoderCore.DecodeImageWithStrips[TPixel](ExifProfile tags, ImageFrame`1 frame, Int32 width, Int32 height, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\Tiff\TiffDecoderCore.cs:line 339
   at SixLabors.ImageSharp.Formats.Tiff.TiffDecoderCore.DecodeFrame[TPixel](ExifProfile tags, Nullable`1 size, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\Tiff\TiffDecoderCore.cs:line 272
   at SixLabors.ImageSharp.Formats.Tiff.TiffDecoderCore.Decode[TPixel](BufferedReadStream stream, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\Tiff\TiffDecoderCore.cs:line 176
   at SixLabors.ImageSharp.Formats.ImageDecoderCore.Decode[TPixel](Configuration configuration, Stream stream, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\ImageDecoderCore.cs:line 82
   --- End of inner exception stack trace ---
   at SixLabors.ImageSharp.Formats.ImageDecoderCore.Decode[TPixel](Configuration configuration, Stream stream, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\ImageDecoderCore.cs:line 86
   at SixLabors.ImageSharp.Formats.Tiff.TiffDecoder.Decode[TPixel](DecoderOptions options, Stream stream, CancellationToken cancellationToken) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\Tiff\TiffDecoder.cs:line 38
   at SixLabors.ImageSharp.Formats.ImageDecoder.<>c__DisplayClass0_0`1.<Decode>b__0(Stream s) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\ImageDecoder.cs:line 24
   at SixLabors.ImageSharp.Formats.ImageDecoder.<WithSeekableStream>g__PerformActionAndResetPosition|11_0[T](Stream s, Int64 position, <>c__DisplayClass11_0`1&) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\ImageDecoder.cs:line 201
   at SixLabors.ImageSharp.Formats.ImageDecoder.WithSeekableStream[T](DecoderOptions options, Stream stream, Func`2 action)
   at SixLabors.ImageSharp.Formats.ImageDecoder.Decode[TPixel](DecoderOptions options, Stream stream) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Formats\ImageDecoder.cs:line 21
   at SixLabors.ImageSharp.Image.Decode[TPixel](DecoderOptions options, Stream stream) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Image.Decode.cs:line 185
   at SixLabors.ImageSharp.Image.<>c__DisplayClass91_0`1.<Load>b__0(Stream s) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Image.FromStream.cs:line 234
   at SixLabors.ImageSharp.Image.WithSeekableStream[T](DecoderOptions options, Stream stream, Func`2 action) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Image.FromStream.cs:line 301
   at SixLabors.ImageSharp.Image.Load[TPixel](DecoderOptions options, Stream stream) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Image.FromStream.cs:line 234
   at SixLabors.ImageSharp.Image.Load[TPixel](DecoderOptions options, String path) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Image.FromFile.cs:line 254
   at SixLabors.ImageSharp.Image.Load[TPixel](String path) in D:\Dev\ImageSharpTest\ImageSharp\src\ImageSharp\Image.FromFile.cs:line 233
   at SixLabors.ImageSharp.Tests.ProfilingSandbox.Program.Main(String[] args) in D:\Dev\ImageSharpTest\ImageSharp\tests\ImageSharp.Tests.ProfilingSandbox\Program.cs:line 43

The presence of negative numbers for the allocation length leads me to believe there is some kind of wrapping/overflow issue.

[JimBobSquarePants]

Yeah, we're attempting to allocate far too large a buffer there. The strip decoder is suffering from the same issue we fixed for tiles in #2874

Can you please raise an issue. Thanks!

[mfeemster]

I've created an issue here: #2959