New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Environment variable for --memory #4028
Comments
We could easily solve this issue by removing any limitation (or selecting a larger limit) for the Window Size at decompression time. While I'm sure it would better fit your use case, the problem is, in other environments, this could be considered a DoS attack vector: a crafted input could instruct the decompressor to allocate a vast amount of memory, with potentially some nasty consequences (we mitigate the worst of it by only reserving the memory address space, and the physical memory itself is normally not used until this amount of data is actually produced, but still). Our proposed solution is to offer a flag so that users can explicitly "opt-in" to employ more memory on their system when they want to. But it seems this solution is not appropriate for your use case. Now your proposition would be to change the default behavior, but as you can guess, this is always going to be in tension with other environments which would prefer other defaults. The next solution that comes to mind is to provide some control through environment variables. |
An environment variable for memory would be great too! I didn't know about the DoS attack vector, thanks for noting that. I'll go ahead and edit the tile and comment to be an FR to be for an environment variable instead. |
Is your feature request related to a problem? Please describe.
Sometimes I use tools that utilize zstd, but they don't allow me to pass arbitrary flags to zstd (in this case
rg -z
). In these cases, the tool won't work in cases where the archive requires additional flags to decompress (such as needing--long=32
or--memory=2048MB
)Describe the solution you'd like
It would be great if there was an environment variable like ZSTD_MEMORY or similar to specify window size for compression/decompression.
This would allow me to use tools where I'm not able to pass options to zstd to decompress files that would need a flag like
--long=32
or--memory=2048MB
.Describe alternatives you've considered
In this case, I wrote a shell script using
xargs
that ranzstdcat --long=31
on files and piped them torg
.Additional context
In this case, I was using
rg
to gather statistics on a large compressed text corpora. Usuallyrg
is nice for search a lot of compressed files in a directory, but these had been compressed with --long, so I got an error[1].I also opened a bug in the ripgrep repo to support passing flags to the compression libraries it uses. That said, this would be a really nice general feature of zstd.
Thank you! :)
Footnotes
[1] Error message from rg, bubbling up an error from zstd:
The text was updated successfully, but these errors were encountered: