Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
feat: Use snappy to compress large messages #859
BREAKING CHANGE: protocol communication break
On the test net monitoring, the bandwidth usage is often in a full state. We try to use the snappy compression algorithm to reduce network transmission consumption.
After testing, the compression yield of flatbuffer format is very high, cpu consumption is relatively acceptable.
The following is the data transmission on the test net:
Note that this is a break change, the data is modified as follows:
By default, data above 40k enters compressed mode.
From the current point of view, the high bit 1 is the compressed format and the high bit 0 is the uncompressed format.
If you want to support multiple compression formats in the future, you can simply think that 0b1000 is in snappy format and 0b0000 is in uncompressed format.
I adjusted the compression threshold: from 40kb to 1kb.
The 20m bandwidth transmission speed is 2.5 m/s, and the 1 kb data requires 0.4 ms.
Snappy compression is 500m/s, our data compression rate is 2x and above, 500b requires 0.2 ms transmission, but the compression time of 1kb to 500b is only 0.002 ms.
The data may be jittered when considering the rate to test, but all cases are 2-5 times lower in compression speed, which does not affect the overall bandwidth utilization. Especially our data compression effect may be much larger than 50%.
However, compression is the time to squeeze the CPU, and we should not compress all the data, which will affect the execution of other tasks.
So, at present, I set the target of compression to be more than 1kb. This threshold can be adjusted in the future. We can watch while walking.