-
Notifications
You must be signed in to change notification settings - Fork 579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable compression in netty #8486
Comments
Go for it, it seems like an easy, low hanging fruit. We would need to pick the right compression algo. Common wisdom is gzip, as it has a very low memory impact, but it doesn't have the best compression/decompression times and/or ratio. Snappy is the one I usually see mentioned for streaming use cases, since it has low memory and cpu impact (at the cost of worse compression). But I guess GZip is usually pretty balanced. I'm not sure how these compare to Zlib, Zstd, or Lzx tbh. I would propose GZip if only because it's balanced and AFAIK the most used for TCP/HTTP compression, but I'm happy to allocate time for benchmarking it compared to other algos which have different concerns (e.g. lz4 or snappy which are optimized for fast compression/decompression at the loss of ratio) |
We can also make it configurable and make more than one algo available. I think, for us optimizing for time/cpu/memory is more important than the compression ratio. |
Then I would propose offering GZip and Snappy. Both don't require much configuration (as opposed to, say, Brotli), one is focused on compression ratio and the other low resource usage. Of course we can discuss other alternatives (e..g I'm not familiar with the difference between lz4 and Snappy, both have similar focuses). |
I think zstd would be a strong contender as well, maybe even instead of gzip. It should be faster than gzip with the same compression ratio. |
I have added Gzip and Snappy in #8502 . We can add zstd as a follow up. Right now, I don't see any performance impact when using Gzip. So I don't know if we will see any difference when using zstd. Besides I had already started testing with Gzip before your comment @oleschoenburg |
Did you measure how the CPU usage changes with and without compression? Are we penalizing users with low network latency, i.e. running all brokers in the same data center, if we enable this by default? |
Okay, I saw that it's optional. Do you have any feeling out of your benchmarks at what network latency it's worth the CPU investment? |
IIRC, Deepthi/Ole didn't notice any performance impact when it comes to either gzip or snappy. That said, Snappy is specifically available for the use case where you want a bit of compression for very little overhead, as that's what it was designed for. So users worried about it but who may face a bit of latency could use Snappy. I would recommend to start with GZIP in general, and if you see no different stick to it, as it gives the best compression. If there is some impact, go to Snappy, and if you're sure latency is great then just keep it disabled. |
Is your feature request related to a problem? Please describe.
More and more usecases are coming up where Zeebe is deployed across multiple data centers. That means, the communication latency between two zeebe brokers can be high.
When there is high latency between nodes, we have previously observed
While we expect that higher latency will impact the commit latency and this overall process execution time and throughput, frequent leader change is not acceptable. One way to prevent leader change was to increase the election timeout. This means that failover time is also higher, as it take longer to start an election when the current leader dies.
One of the causes for the frequent leader change is that the bandwidth limitation of TCP when the RTT is higher.
![image](https://user-images.githubusercontent.com/1997478/147547308-0a6d1e38-3ddb-4943-8d47-3256a2fc9a87.png)
https://accedian.com/blog/measuring-network-performance-latency-throughput-packet-loss/
Since we are sending a lot of data over network, the latency has a big impact on replication throughput. This cause a lot of requests to timeout and this causes a leader election.
Describe the solution you'd like
We can enable compression by adding the following lines when creating a channel (
BasicServer/ClientChannelInitializer#initChannel
There are also other compression algorithms available.
Describe alternatives you've considered
Additional context
See comment for the benchmark result from a prototype.
The text was updated successfully, but these errors were encountered: