*: add a variable to control the back off time and disable txn auto retry by default #10266

jackysp · 2019-04-25T07:06:13Z

What problem does this PR solve?

If TiKV doesn't return a retryable error, TiDB should not retry the transaction. But we need a way to control the back off time to avoid the SQL failing when the PD/TiKV is not available.

What is changed and how it works?

Add a variable to control the back off time, and disable txn auto retry by default.

Check List

Tests

Unit test

Code changes

Has exported function/method change

Side effects

Increased code complexity

jackysp · 2019-04-25T07:06:43Z

/run-all-tests

jackysp · 2019-04-25T09:17:09Z

/rebuild

zhouqiang-cl · 2019-04-25T14:25:43Z

/rebuild

zhouqiang-cl · 2019-04-25T14:26:05Z

/run-all-tests

codecov · 2019-04-29T03:43:25Z

Codecov Report

Merging #10266 into master will decrease coverage by 0.0138%.
The diff coverage is 77.7777%.

@@               Coverage Diff                @@
##             master     #10266        +/-   ##
================================================
- Coverage   77.8343%   77.8204%   -0.0139%     
================================================
  Files           410        410                
  Lines         84306      84330        +24     
================================================
+ Hits          65619      65626         +7     
- Misses        13797      13807        +10     
- Partials       4890       4897         +7

codecov · 2019-04-29T03:43:25Z

Codecov Report

Merging #10266 into master will decrease coverage by 0.0102%.
The diff coverage is 76.923%.

@@               Coverage Diff                @@
##             master     #10266        +/-   ##
================================================
- Coverage   77.3234%   77.3131%   -0.0103%     
================================================
  Files           412        412                
  Lines         85542      85543         +1     
================================================
- Hits          66144      66136         -8     
- Misses        14383      14387         +4     
- Partials       5015       5020         +5

disksing · 2019-04-29T05:32:46Z

I think you can split it into 2 PRs. One for the weight variable, one for refactor write conflict error.

jackysp · 2019-04-29T06:08:50Z

It is not a big PR. I think it is not necessary.

AndreMouche

Should we split this PR into two? It seems we both add tidb_back_off_weight and fix the retryableMark for network error?

kv/variables.go

AndreMouche · 2019-04-29T12:09:42Z

store/tikv/backoff.go

@@ -241,6 +241,9 @@ func (b *Backoffer) WithVars(vars *kv.Variables) *Backoffer {
 	if vars != nil {
 		b.vars = vars
 	}
+	if math.MaxInt32/b.vars.BackOffWeight >= b.maxSleep {
+		b.maxSleep *= b.vars.BackOffWeight


So all backoff will use this weight? How about specifying only prewrite,commit,get_tso which will only be used in 2pc?

What are the benefits of doing this?

I mean, if there is a physical failure, the need to change the backoff should be similar. Still keeping other values fixed, which may make this parameter very useless.

Signed-off-by: Yu Shuaipeng <jackysp@gmail.com>

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

jackysp · 2019-05-05T11:29:37Z

/run-all-tests

jackysp · 2019-05-05T13:24:09Z

/run-common-test tidb-test=pr/800

jackysp · 2019-05-05T13:29:54Z

/run-common-test tidb-test=pr/800

AndreMouche

LGTM

jackysp · 2019-05-06T09:57:48Z

PTAL @disksing @lysu

lysu · 2019-05-06T13:09:51Z

@jackysp Please add more description or comment about background & how-to-use for BackOffWeight, it hard to understand, why we should

	if math.MaxInt32/b.vars.BackOffWeight >= b.maxSleep {
		b.maxSleep *= b.vars.BackOffWeight
	}

and how and when to choose right BackOffWeight value for new viewer like me :D

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

lysu

LGTM

disksing · 2019-05-07T06:36:43Z

LGTM

disksing · 2019-05-07T06:38:55Z

/run-all-tests

zz-jason · 2019-05-07T08:03:28Z

kv/variables.go

@@ -18,6 +18,9 @@ type Variables struct {
 	// BackoffLockFast specifies the LockFast backoff base duration in milliseconds.
 	BackoffLockFast int

+	// BackOffWeight specifies the weight of the max back off time duration.
+	BackOffWeight int


backoff weight is a little confusion, how about directly set the max backoff time?

There are many kinds of backoffs. Maybe set them separately in the future.

jackysp added status/DNM sig/transaction SIG:Transaction labels Apr 25, 2019

jackysp requested review from AndreMouche and disksing April 25, 2019 07:06

jackysp added the status/all tests passed label Apr 26, 2019

AndreMouche reviewed Apr 29, 2019

View reviewed changes

shenli mentioned this pull request May 4, 2019

session: make tidb_disable_txn_auto_retry disable retry for the explicit transactions #10339

Merged

shenli added the Priority/P1 Features that will be implemented in the latest or next major/minor version label May 4, 2019

jackysp added 3 commits May 5, 2019 15:20

*: add a variable to control the back off time

c00aece

change the default value to 2

aed51a8

Signed-off-by: Yu Shuaipeng <jackysp@gmail.com>

address comments

f525557

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

jackysp force-pushed the control_backoff branch from 47fcc9f to f525557 Compare May 5, 2019 07:21

jackysp changed the title *: add a variable to control the back off time *: add a variable to control the back off time and disable txn auto retry by default May 5, 2019

disable auto retry by default

4128691

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

jackysp force-pushed the control_backoff branch from f22e526 to 4128691 Compare May 5, 2019 11:25

AndreMouche previously approved these changes May 6, 2019

View reviewed changes

jackysp removed the status/DNM label May 6, 2019

jackysp requested a review from lysu May 6, 2019 09:57

lysu added the status/LGT1 Indicates that a PR has LGTM 1. label May 6, 2019

add comments

c88a0c5

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

jackysp dismissed AndreMouche’s stale review via c88a0c5 May 6, 2019 13:42

lysu reviewed May 7, 2019

View reviewed changes

lysu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 7, 2019

disksing added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels May 7, 2019

disksing approved these changes May 7, 2019

View reviewed changes

Merge branch 'master' into control_backoff

94b41bd

zz-jason reviewed May 7, 2019

View reviewed changes

Merge branch 'master' into control_backoff

587598e

jackysp merged commit d8589df into pingcap:master May 8, 2019

jackysp deleted the control_backoff branch May 29, 2019 07:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add a variable to control the back off time and disable txn auto retry by default #10266

*: add a variable to control the back off time and disable txn auto retry by default #10266

jackysp commented Apr 25, 2019 •

edited

Loading

jackysp commented Apr 25, 2019

jackysp commented Apr 25, 2019

zhouqiang-cl commented Apr 25, 2019

zhouqiang-cl commented Apr 25, 2019

codecov bot commented Apr 29, 2019

codecov bot commented Apr 29, 2019 •

edited

Loading

disksing commented Apr 29, 2019

jackysp commented Apr 29, 2019

AndreMouche left a comment

AndreMouche Apr 29, 2019

jackysp Apr 29, 2019

jackysp Apr 29, 2019

jackysp commented May 5, 2019

jackysp commented May 5, 2019

jackysp commented May 5, 2019

AndreMouche left a comment

jackysp commented May 6, 2019

lysu commented May 6, 2019

lysu left a comment

disksing commented May 7, 2019

disksing commented May 7, 2019

zz-jason May 7, 2019

jackysp May 7, 2019

*: add a variable to control the back off time and disable txn auto retry by default #10266

*: add a variable to control the back off time and disable txn auto retry by default #10266

Conversation

jackysp commented Apr 25, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

jackysp commented Apr 25, 2019

jackysp commented Apr 25, 2019

zhouqiang-cl commented Apr 25, 2019

zhouqiang-cl commented Apr 25, 2019

codecov bot commented Apr 29, 2019

Codecov Report

codecov bot commented Apr 29, 2019 • edited Loading

Codecov Report

disksing commented Apr 29, 2019

jackysp commented Apr 29, 2019

AndreMouche left a comment

Choose a reason for hiding this comment

AndreMouche Apr 29, 2019

Choose a reason for hiding this comment

jackysp Apr 29, 2019

Choose a reason for hiding this comment

jackysp Apr 29, 2019

Choose a reason for hiding this comment

jackysp commented May 5, 2019

jackysp commented May 5, 2019

jackysp commented May 5, 2019

AndreMouche left a comment

Choose a reason for hiding this comment

jackysp commented May 6, 2019

lysu commented May 6, 2019

lysu left a comment

Choose a reason for hiding this comment

disksing commented May 7, 2019

disksing commented May 7, 2019

zz-jason May 7, 2019

Choose a reason for hiding this comment

jackysp May 7, 2019

Choose a reason for hiding this comment

jackysp commented Apr 25, 2019 •

edited

Loading

codecov bot commented Apr 29, 2019 •

edited

Loading