Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement write barrier for grouping multiple writers #116

Merged
merged 7 commits into from
Oct 14, 2021

Conversation

tabokie
Copy link
Member

@tabokie tabokie commented Oct 14, 2021

Signed-off-by: tabokie xy.tao@outlook.com

When testing TiKV with raft engine, I have observed noticeable performance regression (25%) under low pressure when setting parallelize_fsync to false. The most convincing cause for it is inefficient batching triggered by prolonged latency of one single request (batch size doesn't grow in ratio to latency).

This PR implements a more intelligent batching system at raft engine, similar to RocksDB's write group.

stress benchmark shows nearly 40% improvements when using 4 threads, compared to fsync inside mutex:

> Commit 745e494
Note: switching to '745e494'.
- Thread 1
[write]
Throughput(QPS) = 13145.46
Latency(μs) min = 23, avg = 56.58, p50 = 43, p90 = 106, p95 = 130, p99 = 178, p99.9 = 357, max = 7007
Fairness = 100.0%
Write Bandwidth = 9.8MiB/s
- Thread 2
[write]
Throughput(QPS) = 23425.38
Latency(μs) min = 25, avg = 66.62, p50 = 54, p90 = 113, p95 = 128, p99 = 167, p99.9 = 283, max = 7623
Fairness = 99.9%
Write Bandwidth = 17.2MiB/s
- Thread 4
[write]
Throughput(QPS) = 24017.14
Latency(μs) min = 27, avg = 147.84, p50 = 144, p90 = 211, p95 = 247, p99 = 362, p99.9 = 729, max = 9895
Fairness = 99.8%
Write Bandwidth = 17.9MiB/s
> Branch write-group (current)
Previous HEAD position was 745e494 Avoid warnings when building on mac (#113)
Switched to branch 'write-group'
- Thread 1
[write]
Throughput(QPS) = 13930.66
Latency(μs) min = 24, avg = 53.43, p50 = 39, p90 = 98, p95 = 118, p99 = 161, p99.9 = 252, max = 4017
Fairness = 100.0%
Write Bandwidth = 10.4MiB/s
- Thread 2
[write]
Throughput(QPS) = 23193.29
Latency(μs) min = 25, avg = 67.24, p50 = 55, p90 = 111, p95 = 124, p99 = 160, p99.9 = 293, max = 4651
Fairness = 100.0%
Write Bandwidth = 17.3MiB/s
- Thread 4
[write]
Throughput(QPS) = 32974.90
Latency(μs) min = 27, avg = 102.94, p50 = 107, p90 = 142, p95 = 155, p99 = 194, p99.9 = 347, max = 4951
Fairness = 99.5%
Write Bandwidth = 25.5MiB/s

Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie tabokie merged commit f53092a into tikv:master Oct 14, 2021
@tabokie tabokie deleted the write-group branch October 14, 2021 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant