zstd: encode+decode performance improvements #1869
Conversation
ghost
added
the
There was a problem hiding this comment.
LGTM
I wonder if you could provide the results of any local benchmark, at least for document the improvements.
This will always initialize a new zstd enc/dec even if zstd is not used, but I don't think that will cause a memory issue in any app.
I was reading the other encoders initialization, and it looks for gzip and lz4 is using a sync.Pool https://github.com/Shopify/sarama/blob/65f0fec86aabe011db77ad641d31fddf14f3ca41/compress.go#L14
not sure TBH why that was needed and why snappy or zstd is not using that pattern.
|
@wyndhblb please can you sign the CLA and let us know so we can re-run the status check? |
|
You may also need your commit username+e-mail to be registered against your GitHub account too, it looks like they're not currently? |
dnwe
changed the title
|
Hey @wyndhblb 👋🏼 Wanted to do a friendly follow-up and see if you could sign the CLA, so we could get this merged? 🙏🏼 |
|
@bai i have a long long time ago. |
|
@wyndhblb we're getting this issue from the CLA check
|
the lock in that once per decode/encode is expensive, so just do it once at init Signed-off-by: bo <bo.blanton@gmail.com>
the lock in that once per decode/encode is expensive, so just do it once at init
In an empirical test, on a laptop of course, this gained about a few hundred msg/sec in both consumption/production