Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstandard support in input files #993

Open
bhearsum opened this issue May 24, 2023 · 0 comments
Open

zstandard support in input files #993

bhearsum opened this issue May 24, 2023 · 0 comments

Comments

@bhearsum
Copy link

Feature description

zstandard is a modern compression format that is optimized for size and decompression speed. In both these metrics it outperforms gzip/zlib (at the cost of slower compression times). When working with large models is a good candidate as an intermediate format due to these qualities.

It would be wonderful if marian could support natively decompressing these archives as it already does for gzip.

Citations on zstd vs gzip:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant