Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance and Compression level support #17

Open
dangerousplay opened this issue Nov 21, 2019 · 0 comments
Open

Performance and Compression level support #17

dangerousplay opened this issue Nov 21, 2019 · 0 comments

Comments

@dangerousplay
Copy link
Contributor

Hello, i'm rewriting a Docker logger driver for Gelf format supporting multiline. I have done it in Golang but after some issues with performance (Previous version used compression and tries to parse Json from the log).

I'm trying to bring a support for compression level in Gzip and Zlib, using libdeflater, the only downside is that for each thread for parallel processing needs its own compressor object as described in Support for parallel processing #40

DEFLATE (and zlib and gzip) streams aren't suitable for parallel decompression.

However, if you aren't locked into a data format that uses a single stream, you can easily parallelize at the application layer by dividing the data into chunks before compression, then compressing and/or decompressing the chunks in parallel. libdeflate already works fine for this; just make sure to allocate a separate libdeflate_compressor or libdeflate_decompressor for each concurrent thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant