Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot compress file that larger than 2GB #19

Closed
kiddliu opened this issue Apr 23, 2024 · 2 comments
Closed

Cannot compress file that larger than 2GB #19

kiddliu opened this issue Apr 23, 2024 · 2 comments

Comments

@kiddliu
Copy link

kiddliu commented Apr 23, 2024

It seems that the file about to compress is loaded with API readFile which limits by the Node.js buffer

@dzek69
Copy link
Owner

dzek69 commented Apr 23, 2024

Hello, thanks for your report, this has been fixed... at least partially.

Because the zlib module expects the Buffer (or equivalent data format) and can't work with streams - files must be all loaded into memory. I was able to avoid readFile on too big files and bump the max file size to 4GB (on modern systems), but this is a hard limit I can't currently work around.

I added some warnings into README to address than.

I hope this helps with your use case :)

@kiddliu
Copy link
Author

kiddliu commented Apr 25, 2024

That's OK...now I workaround this with GNU split, feeding each part to the tool.

@kiddliu kiddliu closed this as completed Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants