Arithmetic operation resulted in an overflow.

[sgf]

System.OverflowException
HResult=0x80131516
Message=Arithmetic operation resulted in an overflow.
Source=EasyCompressor.LZ4
StackTrace:
at EasyCompressor.LZ4Compressor.BaseDecompress(Byte[] compressedBytes) in D:\a\EasyCompressor\EasyCompressor\src\EasyCompressor.LZ4\LZ4Compressor.cs:line 42
at EasyCompressor.BaseCompressor.Decompress(Byte[] compressedBytes) in D:\a\EasyCompressor\EasyCompressor\src\EasyCompressor\Compressor\BaseCompressor.cs:line 80
at ConsoleAppCompressTest.Program.Main(String[] args) in G:\Src\my\M3\Mir3Apps\ConsoleAppCompressTest\Program.cs:line 15

            LZ4Compressor lz4 = new LZ4Compressor(level: K4os.Compression.LZ4.LZ4Level.L09_HC);
            var bytes = File.ReadAllBytes(@"F:\Users\super\Desktop\M-Hum.wil");
            Console.WriteLine($"OrgLen:{bytes.Length}");

            var cbytes = lz4.Compress(bytes);
            var dltime = DateTime.Now;
            var dldata = lz4dec.Decompress(cbytes); //thow exception here
            var dllen = dldata.Length;
            Console.WriteLine($"LZ4 HC-9 DecLen:{cbytes.Length}");
            Console.WriteLine($"LZ4 HC-9 Rato:{MathF.Round((float)cbytes.Length / bytes.Length, 2)}");
            Console.WriteLine($"Zstd lvl-1 TakeTime:{DateTime.Now.Subtract(dltime).TotalSeconds}");

thow exception when Decompress

when im testing,the LZ4 very slow(all of Compress And Decompress).
Maybe not fast/good enough for the implementation of https://github.com/MiloszKrajewski/K4os.Compression.LZ4 (Further improvement is needed.)
or maybe better chose: https://github.com/MiloszKrajewski/lz4net

[mjebrahimi]

I found that sometimes LZ4 throws an exception when decompressing large files.
In this case, depending on the file size, one of these errors may occur:

• OverflowException: Arithmetic operation resulted in an overflow
• ArgumentOutOfRangeException: Specified argument was out of the range of valid values

Basically this is a problem with the LZ4 itself and has nothing to do with EasyCompressor and I do not know the reason for this.
I have opened this issue to follow this case.

The MiloszKrajewski/lz4net repo is no longer maintained and has been replaced by MiloszKrajewski/K4os.Compression.LZ4 and This LZ4Compressor just is a wrapper on it and nothing else.

[mjebrahimi]

Relevant issues:
MiloszKrajewski/K4os.Compression.LZ4#42
MiloszKrajewski/K4os.Compression.LZ4#43

[MiloszKrajewski]

@sgf @mjebrahimi

Basically this is a problem with the LZ4 itself and has nothing to do with EasyCompressor and I do not know the reason for this.

I disagree. There is no problem with LZ4 (in this respect), the problem is in your library.

I found that sometimes LZ4 throws an exception when decompressing large files.
In this case, depending on the file size, one of these errors may occur:

• OverflowException: Arithmetic operation resulted in an overflow
• ArgumentOutOfRangeException: Specified argument was out of the range of valid values

No, it does not. It throws in EasyCompressor.LZ4\LZ4Compressor.cs. To be very specific, in:

protected override byte[] BaseDecompress(byte[] compressedBytes)
{
    var target = new byte[compressedBytes.Length * 255].AsSpan();
    int decoded = LZ4Codec.Decode(compressedBytes, target);
    return target.Slice(0, decoded).ToArray();
}

you allocate compressed length * 255 bytes. This mean that ~8.1 MB (8421504 B to be exact) file is enough to allocate over 2GB of memory. Everything over it will cause excaption: ArgumentOutOfRangeException or OverflowException (depending on compiler settings, what fails first: multiplication arithmetic overflow or memory allocation).

...and the StackTrace of original report confirms that:

StackTrace:
at EasyCompressor.LZ4Compressor.BaseDecompress(Byte[] compressedBytes) in D:\a\EasyCompressor\EasyCompressor\src\EasyCompressor.LZ4\LZ4Compressor.cs:line 42
at EasyCompressor.BaseCompressor.Decompress(Byte[] compressedBytes) in D:\a\EasyCompressor\EasyCompressor\src\EasyCompressor\Compressor\BaseCompressor.cs:line 80
at ConsoleAppCompressTest.Program.Main(String[] args) in G:\Src\my\M3\Mir3Apps\ConsoleAppCompressTest\Program.cs:line 15

(the problem is in EasyCompressor.LZ4Compressor.BaseDecompress

Just try this snippet:

int x = 8421505 * 255;

[mjebrahimi]

@MiloszKrajewski OK, you're right.
So is it right to always use the stream method since the original source size is usually unknown at decompression time?

[MiloszKrajewski]

So is it right to always use the stream method since the original source size is usually unknown at decompression time?

Not really. There are several ways to make size known. Like storing it? While compressing you can allocate 4 byes more and actually store length in 4 first bytes (or 4 last bytes if you wish).

Or you can use LZ4Pickler which was designed to do exactly that?

[mjebrahimi]

Great point.
So I refactored the code to store the actual size length (4 bytes) at first.
Thanks, @MiloszKrajewski.