txn: Support bypass specified locks for reads #5798

MyonKeminta · 2019-11-04T03:30:21Z

Signed-off-by: MyonKeminta MyonKeminta@users.noreply.github.com

What have you changed?

This is a part of supporting large transaction.

When an transaction T1 meets another large transaction T2's lock, after invoking check_txn_status on T2, if T2 is not committed and check_txn_status enlarged T2's min_commit_ts to a value greater than T1.start_ts, then T1 can bypass T2's lock to read earlier value. This PR supports this functionality.

This is a critical change. Please be very careful when reviewing this PR.

What is the type of the changes?

New feature (a change which adds functionality)

How is the PR tested?

Unit test
Integration test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

No

Does this PR affect `tidb-ansible`?

No

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta · 2019-11-04T03:36:12Z

/bench

sre-bot · 2019-11-04T05:23:50Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 9591a488029619de0ac9c724860e7cf33712093e
--- tikv: 086056fabd7f4763d111856854ed18f6ec9bd9da
+++ tikv: bc29f01784ca73aca61497c80f676657215cae34
pd: ba01076ee005b5545d679dd903f2674416176f09
================================================================================
oltp_update_non_index:
    * QPS: 4729.66 ± 0.14% (std=4.23) delta: -0.01% (p=0.950)
    * Latency p50: 27.06 ± 0.15% (std=0.03) delta: 0.01%
    * Latency p99: 42.05 ± 5.03% (std=1.26) delta: 2.32%
            
oltp_insert:
    * QPS: 4701.13 ± 0.38% (std=10.80) delta: 0.26% (p=0.164)
    * Latency p50: 27.22 ± 0.38% (std=0.06) delta: -0.27%
    * Latency p99: 48.34 ± 0.00% (std=0.00) delta: -2.25%
            
oltp_read_write:
    * QPS: 15678.11 ± 0.35% (std=36.77) delta: 0.69% (p=0.014)
    * Latency p50: 163.60 ± 0.34% (std=0.39) delta: -0.69%
    * Latency p99: 312.58 ± 1.20% (std=2.64) delta: -7.93%
            
oltp_update_index:
    * QPS: 4263.71 ± 0.52% (std=14.89) delta: -0.11% (p=0.640)
    * Latency p50: 30.02 ± 0.53% (std=0.11) delta: 0.11%
    * Latency p99: 54.84 ± 3.64% (std=1.22) delta: 1.84%
            
oltp_point_select:
    * QPS: 45425.58 ± 0.59% (std=230.87) delta: -0.09% (p=0.852)
    * Latency p50: 2.81 ± 0.53% (std=0.02) delta: 0.09%
    * Latency p99: 9.06 ± 0.00% (std=0.00) delta: 0.00%

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…MyonKeminta/tikv into misono/read-bypass-checked-lock

…d-bypass-checked-lock

MyonKeminta · 2019-11-05T03:42:26Z

/run-integration-common-test

MyonKeminta · 2019-11-05T03:49:34Z

/bench

MyonKeminta · 2019-11-05T03:51:00Z

/release

sre-bot · 2019-11-05T04:04:19Z

download tikv at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/297ad0249f5044c9e505280cc7e97490c0611652/centos7/tikv-server.tar.gz

MyonKeminta · 2019-11-05T04:11:31Z

/bench

MyonKeminta · 2019-11-05T05:25:41Z

/bench

MyonKeminta · 2019-11-05T05:28:32Z

/release

sre-bot · 2019-11-05T05:41:31Z

download tikv at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/cca15e949c32d94a83d89c943fe7f401ffa75fcb/centos7/tikv-server.tar.gz

MyonKeminta · 2019-11-05T05:56:39Z

I want to run a benchmark to reveal whether there is significant performance regression caused by this PR but it seems the auto benchmark doesn't work...

MyonKeminta · 2019-11-05T06:24:26Z

/bench

sre-bot · 2019-11-05T07:57:10Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 579304039685562d390c0de5b783425efbd2bd8d
--- tikv: da9168bf76e56c60576ab3bb1780afff49d936f3
+++ tikv: cca15e949c32d94a83d89c943fe7f401ffa75fcb
pd: 10a52ee4d92505add7551c6651b1982b705af678
================================================================================
oltp_update_non_index:
    * QPS: 4696.68 ± 0.11% (std=3.75) delta: -0.46% (p=0.327)
    * Latency p50: 27.19 ± 0.64% (std=0.10) delta: 0.27%
    * Latency p99: 43.65 ± 1.19% (std=0.37) delta: 4.93%
            
oltp_insert:
    * QPS: 4771.18 ± 0.05% (std=1.78) delta: -0.01% (p=0.544)
    * Latency p50: 26.82 ± 0.05% (std=0.01) delta: 0.01%
    * Latency p99: 49.89 ± 2.27% (std=0.75) delta: 3.58%
            
oltp_read_write:
    * QPS: 15432.96 ± 0.37% (std=35.16) delta: -0.04% (p=0.407)
    * Latency p50: 166.16 ± 0.37% (std=0.37) delta: 0.14%
    * Latency p99: 315.91 ± 2.24% (std=4.70) delta: 0.00%
            
oltp_update_index:
    * QPS: 4235.23 ± 0.58% (std=15.46) delta: 0.11% (p=0.648)
    * Latency p50: 30.23 ± 0.58% (std=0.11) delta: -0.12%
    * Latency p99: 53.38 ± 2.72% (std=1.08) delta: -0.88%
            
oltp_point_select:
    * QPS: 45167.62 ± 0.41% (std=131.52) delta: -0.67% (p=0.073)
    * Latency p50: 2.83 ± 0.47% (std=0.01) delta: 0.74%
    * Latency p99: 9.39 ± 0.00% (std=0.00) delta: 3.64%

MyonKeminta · 2019-11-05T12:13:28Z

It seems it doesn't harm performance much.

…d-bypass-checked-lock

…MyonKeminta/tikv into misono/read-bypass-checked-lock

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

src/storage/mvcc/mod.rs

Signed-off-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-Authored-By: Lei Zhao <zlwgx1023@gmail.com>

sticnarf · 2019-11-08T02:46:42Z

It seems it doesn't harm performance much.

sysbench is almost oltp. I'd like to try some AP scenarios...

sticnarf · 2019-11-08T02:46:53Z

/bench +tpch

sre-bot · 2019-11-08T08:07:46Z

Benchmark Report

Run TPC-H 10G Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: ccbdfdc5178cb84ec97577714b15624adc95380f
--- tikv: d019ccecefc260ff760a53b7b8742fb84ffca9b5
+++ tikv: 0a1499cc56f6298a567fe77d3d0041083beef913
pd: d6d08d9b58fa42a29c3b3f517b9a64a562475588
================================================================================
01.sql duration: 20210.15 ms ± 0.53% (std=68.27) delta: 4.49% (p=0.115)
02.sql duration: 7328.49 ms ± 4.57% (std=160.34) delta: 0.66% (p=0.193)
03.sql duration: 18411.80 ms ± 2.82% (std=273.98) delta: 2.39% (p=0.006)
04.sql duration: 7912.53 ms ± 1.36% (std=65.48) delta: 2.53% (p=0.008)
06.sql duration: 11909.05 ms ± 3.58% (std=224.18) delta: 12.03% (p=0.000)
07.sql duration: 17131.81 ms ± 2.68% (std=266.14) delta: 15.97% (p=0.000)
08.sql duration: 12972.24 ms ± 1.81% (std=128.05) delta: 5.07% (p=0.000)
09.sql duration: 32710.91 ms ± 0.86% (std=166.34) delta: 1.22% (p=0.004)
10.sql duration: 10579.74 ms ± 5.89% (std=333.27) delta: 3.37% (p=0.024)
11.sql duration: 3818.45 ms ± 5.12% (std=117.67) delta: -7.58% (p=0.026)
12.sql duration: 13495.40 ms ± 7.40% (std=426.07) delta: 8.12% (p=0.000)
13.sql duration: 9123.29 ms ± 1.20% (std=61.70) delta: 0.57% (p=0.176)
14.sql duration: 13322.05 ms ± 3.99% (std=254.67) delta: 11.36% (p=0.000)
15.sql duration: 24703.90 ms ± 6.57% (std=780.01) delta: 9.90% (p=0.000)
16.sql duration: 3703.83 ms ± 10.66% (std=226.33) delta: 2.31% (p=0.613)
17.sql duration: 37312.87 ms ± 1.47% (std=333.02) delta: 1.07% (p=0.672)
18.sql duration: 55925.76 ms ± 6.31% (std=2186.15) delta: -0.31% (p=0.756)
19.sql duration: 16229.23 ms ± 5.88% (std=500.72) delta: -0.22% (p=0.426)
20.sql duration: 13937.53 ms ± 5.78% (std=521.62) delta: 11.71% (p=0.000)
21.sql duration: 34304.90 ms ± 1.44% (std=327.02) delta: 17.58% (p=0.000)
22.sql duration: 5188.18 ms ± 6.52% (std=183.80) delta: 4.24% (p=0.502)

…d-bypass-checked-lock

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

coocood · 2019-11-08T09:06:03Z

LGTM

sticnarf · 2019-11-08T09:07:38Z

/bench +tpch

sre-bot · 2019-11-08T14:47:11Z

Benchmark Report

Run TPC-H 10G Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 87357daba56e5d009653cec4d3996621d436e921
--- tikv: 1999eaf8a6da5d4d7cc8f89a0d1127c516614aef
+++ tikv: aebad246522dfeb05eaf6575dd7449e2f697d38e
pd: d6d08d9b58fa42a29c3b3f517b9a64a562475588
================================================================================
01.sql duration: 18926.16 ms ± 0.22% (std=32.52) delta: -0.49% (p=0.630)
02.sql duration: 7494.42 ms ± 3.27% (std=140.65) delta: -0.46% (p=0.480)
03.sql duration: 17555.67 ms ± 1.57% (std=151.71) delta: -4.00% (p=0.000)
04.sql duration: 7576.70 ms ± 2.65% (std=102.09) delta: 0.56% (p=0.430)
06.sql duration: 10628.32 ms ± 6.69% (std=552.20) delta: 1.90% (p=0.069)
07.sql duration: 14911.16 ms ± 3.17% (std=302.80) delta: 1.50% (p=0.078)
08.sql duration: 12877.82 ms ± 1.66% (std=124.04) delta: 2.70% (p=0.000)
09.sql duration: 33663.63 ms ± 3.03% (std=600.69) delta: 1.34% (p=0.402)
10.sql duration: 10450.32 ms ± 2.79% (std=154.79) delta: 0.04% (p=0.921)
11.sql duration: 3698.14 ms ± 0.98% (std=20.08) delta: -2.10% (p=0.024)
12.sql duration: 12544.98 ms ± 2.93% (std=209.48) delta: 2.64% (p=0.188)
13.sql duration: 9410.92 ms ± 5.63% (std=249.32) delta: 3.43% (p=0.074)
14.sql duration: 12007.14 ms ± 2.70% (std=228.98) delta: 1.91% (p=0.874)
15.sql duration: 23035.28 ms ± 4.37% (std=643.00) delta: 7.49% (p=0.006)
16.sql duration: 3869.06 ms ± 14.43% (std=303.38) delta: 0.70% (p=0.363)
17.sql duration: 38518.79 ms ± 1.42% (std=312.57) delta: 0.15% (p=0.910)
18.sql duration: 57919.08 ms ± 2.02% (std=712.21) delta: -0.15% (p=0.553)
19.sql duration: 15665.32 ms ± 3.25% (std=243.74) delta: -4.25% (p=0.000)
20.sql duration: 12120.41 ms ± 2.66% (std=162.25) delta: 1.45% (p=0.209)
21.sql duration: 30523.10 ms ± 1.10% (std=200.73) delta: 1.59% (p=0.004)
22.sql duration: 5103.51 ms ± 5.77% (std=174.59) delta: 3.80% (p=0.060)

sticnarf · 2019-11-12T04:09:51Z

Actually I think the Arc is avoidable, but it should introduce some lifetimes in ScannerConfig and PointGetter. Probably not that necessary...

MyonKeminta · 2019-11-12T07:45:58Z

@sticnarf I think it's only sometimes avoidable in the current structure, since the SnapshotScanner sometimes needs impl Send to move to another thread.

MyonKeminta · 2019-11-12T07:46:14Z

/release

sre-bot · 2019-11-12T07:59:15Z

download tikv at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/8bf5803d475b1df21ca518ef0f53e2f20f022b1a/centos7/tikv-server.tar.gz

sticnarf

LGTM

sre-bot · 2019-11-12T08:34:54Z

Your auto merge job has been accepted, waiting for 5837, 5843, 5829

sre-bot · 2019-11-12T09:33:01Z

/run-all-tests

sre-bot · 2019-11-12T09:50:29Z

@MyonKeminta merge failed.

youjiali1995 · 2019-11-12T09:52:51Z

/test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Support skip lock for scanner and point getter

5291bc3

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta added sig/transaction SIG: Transaction S: WIP labels Nov 4, 2019

Merge branch 'master' into misono/read-bypass-checked-lock

bc29f01

MyonKeminta changed the title ~~[DNM] txn: Support skip lock for scanner and point getter~~ [DNM] txn: Support bypass specified locks for reads Nov 4, 2019

MyonKeminta added 3 commits November 4, 2019 16:46

Set bypass locks for coprocessor requests

4a09909

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Merge branch 'misono/read-bypass-checked-lock' of https://github.com/…

9aee639

…MyonKeminta/tikv into misono/read-bypass-checked-lock

Merge branch 'master' of https://github.com/tikv/tikv into misono/rea…

297ad02

…d-bypass-checked-lock

Merge branch 'master' into misono/read-bypass-checked-lock

cca15e9

MyonKeminta added 5 commits November 6, 2019 16:14

Merge branch 'master' of https://github.com/tikv/tikv into misono/rea…

bc9863b

…d-bypass-checked-lock

Merge branch 'misono/read-bypass-checked-lock' of https://github.com/…

cafd5df

…MyonKeminta/tikv into misono/read-bypass-checked-lock

Avoid Arc cloning when the set is empty

0aaa867

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Add test for ts set and check_lock

ac5bf68

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Add tests to point getter and scanner

0b2cb6f

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta removed the S: WIP label Nov 7, 2019

MyonKeminta changed the title ~~[DNM] txn: Support bypass specified locks for reads~~ txn: Support bypass specified locks for reads Nov 7, 2019

MyonKeminta marked this pull request as ready for review November 7, 2019 09:03

youjiali1995 reviewed Nov 7, 2019

View reviewed changes

src/storage/mvcc/mod.rs Outdated Show resolved Hide resolved

Update src/storage/mvcc/mod.rs

0a1499c

Signed-off-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-Authored-By: Lei Zhao <zlwgx1023@gmail.com>

MyonKeminta force-pushed the misono/read-bypass-checked-lock branch from 8904693 to 0a1499c Compare November 7, 2019 09:32

MyonKeminta added 2 commits November 8, 2019 16:51

Merge branch 'master' of https://github.com/tikv/tikv into misono/rea…

6637948

…d-bypass-checked-lock

Add an inline to make it looks better

aebad24

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Merge branch 'master' into misono/read-bypass-checked-lock

8bf5803

sticnarf approved these changes Nov 12, 2019

View reviewed changes

youjiali1995 approved these changes Nov 12, 2019

View reviewed changes

youjiali1995 added the status/can-merge Status: Can merge to base branch label Nov 12, 2019

Merge branch 'master' into misono/read-bypass-checked-lock

3fecda3

youjiali1995 merged commit ed4d332 into tikv:master Nov 12, 2019

MyonKeminta deleted the misono/read-bypass-checked-lock branch November 13, 2019 05:27

hawkingrei pushed a commit to hawkingrei/tikv that referenced this pull request Dec 1, 2019

txn: Support bypass specified locks for reads (tikv#5798)

e4f2b6a

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

longfangsong mentioned this pull request Jul 27, 2020

Transaction Handling Newbie Perspective tikv/sig-transaction#12

Merged

longfangsong mentioned this pull request Apr 1, 2021

[Contribute] bypass_locks longfangsong/tipedia#110

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

txn: Support bypass specified locks for reads #5798

txn: Support bypass specified locks for reads #5798

MyonKeminta commented Nov 4, 2019 •

edited

MyonKeminta commented Nov 4, 2019

sre-bot commented Nov 4, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sre-bot commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sre-bot commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sre-bot commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sticnarf commented Nov 8, 2019

sticnarf commented Nov 8, 2019

sre-bot commented Nov 8, 2019

coocood commented Nov 8, 2019

sticnarf commented Nov 8, 2019

sre-bot commented Nov 8, 2019

sticnarf commented Nov 12, 2019

MyonKeminta commented Nov 12, 2019 •

edited

MyonKeminta commented Nov 12, 2019

sre-bot commented Nov 12, 2019

sticnarf left a comment

sre-bot commented Nov 12, 2019

sre-bot commented Nov 12, 2019

sre-bot commented Nov 12, 2019

youjiali1995 commented Nov 12, 2019

txn: Support bypass specified locks for reads #5798

txn: Support bypass specified locks for reads #5798

Conversation

MyonKeminta commented Nov 4, 2019 • edited

What have you changed?

What is the type of the changes?

How is the PR tested?

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

Does this PR affect tidb-ansible?

MyonKeminta commented Nov 4, 2019

sre-bot commented Nov 4, 2019

Benchmark Report

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sre-bot commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sre-bot commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

MyonKeminta commented Nov 5, 2019

sre-bot commented Nov 5, 2019

Benchmark Report

MyonKeminta commented Nov 5, 2019

sticnarf commented Nov 8, 2019

sticnarf commented Nov 8, 2019

sre-bot commented Nov 8, 2019

Benchmark Report

coocood commented Nov 8, 2019

sticnarf commented Nov 8, 2019

sre-bot commented Nov 8, 2019

Benchmark Report

sticnarf commented Nov 12, 2019

MyonKeminta commented Nov 12, 2019 • edited

MyonKeminta commented Nov 12, 2019

sre-bot commented Nov 12, 2019

sticnarf left a comment

Choose a reason for hiding this comment

sre-bot commented Nov 12, 2019

sre-bot commented Nov 12, 2019

sre-bot commented Nov 12, 2019

youjiali1995 commented Nov 12, 2019

MyonKeminta commented Nov 4, 2019 •

edited

Does this PR affect `tidb-ansible`?

MyonKeminta commented Nov 12, 2019 •

edited