Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use snappy to compress large messages #859

Merged
merged 4 commits into from May 24, 2019

Conversation

@driftluo
Copy link
Member

commented May 21, 2019

BREAKING CHANGE: protocol communication break

On the test net monitoring, the bandwidth usage is often in a full state. We try to use the snappy compression algorithm to reduce network transmission consumption.

After testing, the compression yield of flatbuffer format is very high, cpu consumption is relatively acceptable.

The following is the data transmission on the test net:

2019-05-20 16:27:41.875 +08:00 tokio-runtime-worker-7 DEBUG compress  raw_data len: 625400, compress used time: 3.635121ms, compress_data size: 335401, compression ratio: 0.536298369043812, decompress used time: 1.496667ms
2019-05-20 16:27:42.128 +08:00 tokio-runtime-worker-6 DEBUG compress  raw_data len: 633544, compress used time: 3.789752ms, compress_data size: 335462, compression ratio: 0.5295007134468955, decompress used time: 1.490144ms
2019-05-20 16:27:42.340 +08:00 tokio-runtime-worker-6 DEBUG compress  raw_data len: 633216, compress used time: 3.998678ms, compress_data size: 333458, compression ratio: 0.5266101930462906, decompress used time: 1.593165ms
2019-05-20 16:27:42.558 +08:00 tokio-runtime-worker-5 DEBUG compress  raw_data len: 632992, compress used time: 3.453616ms, compress_data size: 333552, compression ratio: 0.5269450482786512, decompress used time: 1.052606ms
2019-05-20 16:27:42.740 +08:00 tokio-runtime-worker-2 DEBUG compress  raw_data len: 633760, compress used time: 1.256847ms, compress_data size: 340022, compression ratio: 0.5365154001514769, decompress used time: 545.473µs
2019-05-20 16:37:43.934 +08:00 tokio-runtime-worker-1 DEBUG compress  raw_data len: 186912, compress used time: 659.317µs, compress_data size: 42640, compression ratio: 0.22812874507789763, decompress used time: 515.287µs
2019-05-20 16:37:47.338 +08:00 tokio-runtime-worker-3 DEBUG compress  raw_data len: 186520, compress used time: 189.079µs, compress_data size: 42334, compression ratio: 0.22696761741368218, decompress used time: 150.644µs
2019-05-20 16:37:50.729 +08:00 tokio-runtime-worker-3 DEBUG compress  raw_data len: 186520, compress used time: 197.656µs, compress_data size: 42336, compression ratio: 0.22697834012438345, decompress used time: 145.5µs
2019-05-20 16:38:52.549 +08:00 tokio-runtime-worker-4 DEBUG compress  raw_data len: 95904, compress used time: 217.968µs, compress_data size: 33801, compression ratio: 0.3524461961961962, decompress used time: 95.818µs
2019-05-20 16:39:32.522 +08:00 tokio-runtime-worker-0 DEBUG compress  raw_data len: 47320, compress used time: 418.183µs, compress_data size: 17183, compression ratio: 0.363123415046492, decompress used time: 252.148µs

Note that this is a break change, the data is modified as follows:

By default, data above 40k enters compressed mode.

From the current point of view, the high bit 1 is the compressed format and the high bit 0 is the uncompressed format.

If you want to support multiple compression formats in the future, you can simply think that 0b1000 is in snappy format and 0b0000 is in uncompressed format.

 # Message in Bytes:

 +---------------------------------------------------------------+
 | Bytes | Type | Function                                       |
 |-------+------+------------------------------------------------|
 |   0   |  u1  | Compress: true 1, false 0                      |
 |       |  u7  | Reserved                                       |
 +-------+------+------------------------------------------------+
 |  1~   |      | Payload (Serialized Data with Compress)        |
 +-------+------+------------------------------------------------+

@driftluo driftluo requested review from TheWaWaR, yangby-cryptape and nervosnetwork/ckb-dev May 21, 2019

@driftluo driftluo requested a review from nervosnetwork/ckb-code-review as a code owner May 21, 2019

@doitian doitian added the b:p2p label May 22, 2019

network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
@driftluo

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

@doitian done

network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved
@driftluo driftluo referenced this pull request May 23, 2019
2 of 2 tasks complete

@driftluo driftluo force-pushed the driftluo:add-compress-feature branch from be739f1 to d4cc0c3 May 23, 2019

@driftluo

This comment has been minimized.

Copy link
Member Author

commented May 23, 2019

I adjusted the compression threshold: from 40kb to 1kb.

The 20m bandwidth transmission speed is 2.5 m/s, and the 1 kb data requires 0.4 ms.

Snappy compression is 500m/s, our data compression rate is 2x and above, 500b requires 0.2 ms transmission, but the compression time of 1kb to 500b is only 0.002 ms.

The data may be jittered when considering the rate to test, but all cases are 2-5 times lower in compression speed, which does not affect the overall bandwidth utilization. Especially our data compression effect may be much larger than 50%.

However, compression is the time to squeeze the CPU, and we should not compress all the data, which will affect the execution of other tasks.

So, at present, I set the target of compression to be more than 1kb. This threshold can be adjusted in the future. We can watch while walking.

@driftluo driftluo requested review from quake and doitian May 24, 2019

network/src/compress.rs Outdated Show resolved Hide resolved
network/src/compress.rs Outdated Show resolved Hide resolved

@driftluo driftluo force-pushed the driftluo:add-compress-feature branch from 922e41a to 973f16c May 24, 2019

@quake

quake approved these changes May 24, 2019

@driftluo driftluo merged commit 52a56f6 into nervosnetwork:develop May 24, 2019

2 checks passed

Nervos CI Build Passed via Travis
Details
Travis CI - Pull Request Build Passed
Details

@driftluo driftluo deleted the driftluo:add-compress-feature branch May 24, 2019

This was referenced May 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.