Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch prewrite and commit requests to same region across batch commands #5756

Merged
merged 65 commits into from Nov 27, 2019

Conversation

@tabokie
Copy link
Member

tabokie commented Oct 29, 2019

What have you changed?

  • Batch prewrite (optimistic) / commit requests to the same region.

What is the type of the changes?

  • Improvement (a change which is an improvement to an existing feature)

How is the PR tested?

  • Unit test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

No

Does this PR affect tidb-ansible?

Will do

Refer to a related PR or issue link (optional)

This PR follows up on #5598

Benchmark result if necessary (optional)

Any examples? (optional)

tabokie added 4 commits Oct 29, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie tabokie force-pushed the tabokie:batch-write branch from cc63aa4 to 8040a15 Oct 30, 2019
let cid = match tasks[0].1 {
Msg::ReadFinished { cid, .. } => cid,
Msg::WriteFinished { cid, .. } => cid,
Msg::FinishedWithErr { cid, .. } => cid,
_ => panic!("msg type mismatch"),
};
self.on_batch_finish(
Comment on lines 554 to 560

This comment has been minimized.

Copy link
@Little-Wallace

Little-Wallace Oct 31, 2019

Contributor

Why use id of task[0]? Should the id of every task be same?

This comment has been minimized.

Copy link
@tabokie

tabokie Oct 31, 2019

Author Member

yep, all sub-commands share the same command id.

This comment has been minimized.

Copy link
@Little-Wallace

Little-Wallace Oct 31, 2019

Contributor

Do all commands in process_batch_write_impl share the same id? But why do you still pass a vector of pair rather than on_batch_msg(task_id: u64, results: Vec<Msg>)?

This comment has been minimized.

Copy link
@tabokie

tabokie Oct 31, 2019

Author Member

those are two different id. command id cid is assigned by scheduler, request id id is used by tidb to identify requests in a stream. I could work out a better name later.

@tikv tikv deleted a comment from sre-bot Oct 31, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie tabokie force-pushed the tabokie:batch-write branch from 4df01dd to b968a74 Oct 31, 2019
@zhouqiang-cl

This comment has been minimized.

Copy link
Member

zhouqiang-cl commented Oct 31, 2019

/run-all-tests

tabokie added 3 commits Oct 31, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@tikv tikv deleted a comment from sre-bot Oct 31, 2019
@tikv tikv deleted a comment from sre-bot Oct 31, 2019
@tikv tikv deleted a comment from sre-bot Oct 31, 2019
tabokie added 6 commits Nov 1, 2019
polishing
Signed-off-by: tabokie <xy.tao@outlook.com>
minor fix
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
bump
Signed-off-by: tabokie <xy.tao@outlook.com>
@tikv tikv deleted a comment from sre-bot Nov 5, 2019
tabokie added 2 commits Nov 5, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
fmt
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Nov 20, 2019

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: e4d199d93a9db4565752220f76c6a921e393e328
--- tikv: 45d07db7aa75bdd9f7f508d34c93f6d140976815
+++ tikv: 66b766e88fb97f7d2d765dd4ff4bacc317453402
pd: c71ec9500e49c933c198bba1f901a5b42a72de86
================================================================================
oltp_update_non_index:
    * QPS: 4852.59 ± 0.16% (std=5.28) delta: 2.95% (p=0.000)
    * Latency p50: 26.37 ± 0.16% (std=0.03) delta: -2.88%
    * Latency p99: 43.60 ± 4.01% (std=1.15) delta: 4.80%
            
oltp_insert:
    * QPS: 4768.46 ± 0.32% (std=10.95) delta: 0.09% (p=0.580)
    * Latency p50: 26.84 ± 0.33% (std=0.06) delta: -0.08%
    * Latency p99: 47.74 ± 7.48% (std=2.27) delta: -0.65%
            
oltp_read_write:
    * QPS: 15758.01 ± 0.20% (std=21.35) delta: 1.14% (p=0.008)
    * Latency p50: 162.71 ± 0.13% (std=0.15) delta: -1.16%
    * Latency p99: 307.00 ± 1.20% (std=2.60) delta: -0.70%
            
oltp_update_index:
    * QPS: 4338.80 ± 3.17% (std=83.88) delta: 1.93% (p=0.188)
    * Latency p50: 29.52 ± 3.27% (std=0.59) delta: -1.85%
    * Latency p99: 53.13 ± 2.24% (std=0.79) delta: -1.93%
            
oltp_point_select:
    * QPS: 46604.06 ± 0.08% (std=27.12) delta: 2.44% (p=0.004)
    * Latency p50: 2.75 ± 0.24% (std=0.00) delta: -2.02%
    * Latency p99: 9.02 ± 2.22% (std=0.13) delta: -0.44%
            
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
@tikv tikv deleted a comment from sre-bot Nov 21, 2019
tabokie added 2 commits Nov 21, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie

This comment has been minimized.

Copy link
Member Author

tabokie commented Nov 21, 2019

/bench

@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Nov 21, 2019

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 567f0d485ec4bcfa391441aa962c2b748e825cad
--- tikv: 6fff85c2fde0cf95ecea4956f6ab15a3f722b564
+++ tikv: f29f965f0c1b5baad731c6c8fe3be02210eacb0e
pd: c71ec9500e49c933c198bba1f901a5b42a72de86
================================================================================
oltp_update_non_index:
    * QPS: 4852.49 ± 0.23% (std=8.24) delta: 2.95% (p=0.000)
    * Latency p50: 26.38 ± 0.24% (std=0.04) delta: -2.86%
    * Latency p99: 42.27 ± 4.49% (std=1.90) delta: -1.29%
            
oltp_insert:
    * QPS: 4763.06 ± 0.19% (std=8.00) delta: -0.16% (p=0.131)
    * Latency p50: 26.87 ± 0.19% (std=0.05) delta: 0.16%
    * Latency p99: 47.28 ± 4.87% (std=1.38) delta: 2.75%
            
oltp_read_write:
    * QPS: 15697.87 ± 0.38% (std=35.78) delta: 1.27% (p=0.002)
    * Latency p50: 163.21 ± 0.06% (std=0.07) delta: -1.30%
    * Latency p99: 324.08 ± 2.42% (std=5.53) delta: 5.35%
            
oltp_update_index:
    * QPS: 4383.27 ± 0.77% (std=20.83) delta: 2.99% (p=0.001)
    * Latency p50: 29.20 ± 0.78% (std=0.14) delta: -2.88%
    * Latency p99: 53.61 ± 2.27% (std=0.80) delta: 0.75%
            
oltp_point_select:
    * QPS: 46124.38 ± 0.41% (std=136.49) delta: 1.67% (p=0.029)
    * Latency p50: 2.77 ± 0.48% (std=0.01) delta: -1.65%
    * Latency p99: 9.06 ± 0.00% (std=0.00) delta: -1.33%
            
@tabokie

This comment has been minimized.

Copy link
Member Author

tabokie commented Nov 21, 2019

/bench

workload:
  threads: 1024
@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Nov 21, 2019

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 96069cd74957dc998516f45932d55f7db924cc67
--- tikv: 84986fc2fcef65a07c51892fb1f0c04e0038969f
+++ tikv: f29f965f0c1b5baad731c6c8fe3be02210eacb0e
pd: c71ec9500e49c933c198bba1f901a5b42a72de86
================================================================================
oltp_update_non_index:
    * QPS: 15465.84 ± 0.45% (std=48.52) delta: 3.31% (p=0.000)
    * Latency p50: 66.18 ± 0.45% (std=0.21) delta: -3.21%
    * Latency p99: 148.95 ± 0.90% (std=1.34) delta: -0.89%
            
oltp_insert:
    * QPS: 13377.23 ± 0.44% (std=43.30) delta: 9.56% (p=0.000)
    * Latency p50: 76.52 ± 0.44% (std=0.25) delta: -8.73%
    * Latency p99: 137.35 ± 0.00% (std=0.00) delta: -3.54%
            
oltp_read_write:
    * QPS: 16254.91 ± 0.58% (std=63.32) delta: 0.97% (p=0.015)
    * Latency p50: 1280.23 ± 0.63% (std=5.16) delta: -1.00%
    * Latency p99: 2479.03 ± 1.20% (std=20.98) delta: -2.37%
            
oltp_update_index:
    * QPS: 9617.03 ± 0.61% (std=46.26) delta: 3.52% (p=0.000)
    * Latency p50: 105.12 ± 0.68% (std=0.50) delta: -3.68%
    * Latency p99: 223.34 ± 0.00% (std=0.00) delta: -1.20%
            
oltp_point_select:
    * QPS: 55770.71 ± 0.85% (std=437.43) delta: -0.80% (p=0.279)
    * Latency p50: 18.35 ± 0.83% (std=0.14) delta: 0.82%
    * Latency p99: 48.77 ± 0.89% (std=0.43) delta: 0.90%
            
Signed-off-by: tabokie <xy.tao@outlook.com>
@Little-Wallace

This comment has been minimized.

Copy link
Contributor

Little-Wallace commented Nov 21, 2019

/test

tabokie added 3 commits Nov 21, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: tabokie <xy.tao@outlook.com>
Copy link
Contributor

Little-Wallace left a comment

LGTM

@Little-Wallace

This comment has been minimized.

Copy link
Contributor

Little-Wallace commented Nov 22, 2019

@Little-Wallace Little-Wallace changed the base branch from master to perf-4.0 Nov 27, 2019
@Little-Wallace Little-Wallace merged commit ec13a21 into tikv:perf-4.0 Nov 27, 2019
2 of 3 checks passed
2 of 3 checks passed
idc-jenkins-ci-tikv/integration-common-test Jenkins job failed
Details
DCO All commits are signed off!
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
Little-Wallace added a commit to Little-Wallace/tikv that referenced this pull request Dec 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.