Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ArgumentOutOfRangeException Occurs when decompressing large files #43

Closed
mjebrahimi opened this issue Sep 12, 2020 · 1 comment
Closed
Labels
invalid This doesn't seem right

Comments

@mjebrahimi
Copy link

This exception occurs when decompressing files with actual large size (for example 900 MB).
To reproduce you can use this sample with a large file on your machine and I hope this error occurs to you too ;)

@MiloszKrajewski
Copy link
Owner

MiloszKrajewski commented Sep 12, 2020

It blows up in test, not in compression (source.Length * 255) is too large to allocate.
There are many things wrong with this example you gave. I mean I don't know which parts are just 'for example' and where you think it is the right way to use it, so:

  • for large amount of data you should you use LZ4Stream
  • you should know how much data you expect after decompression when using blocks (LZ4Codec), it is very ineffective to use length * 255
  • if you really dont want to use LZ4Steam and you don't want to implement your own block "packaging" use LZ4Pickler.
  • array.Take(length).ToArray() is one of the slowest ways to get a slice of array (this might be 10 slower then compression itself); you can use it for testing, but please don't use it for performance testing (array.AsSpan(0, length).ToArray() is not that bad)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invalid This doesn't seem right
Projects
None yet
Development

No branches or pull requests

2 participants