pessimistic-txn: solve non-pessimistic-lock conflict #4801

youjiali1995 · 2019-05-29T05:29:32Z

What have you changed? (mandatory)

Upgrade to the latest kvproto, which added for_update_ts field to prewrite request.
Record for_update_ts in Lock, and judge whether a transaction is optimistic or pessimistic by it.
Add for_update_ts field to Options to avoid changing prewrite interface, and change acquire_pessimistic_lock interface to adapt it.
Overwrite optimistic lock in pessimistic_prewrite if request's for_update_ts is greater than lock's for_update_ts.

What are the type of the changes? (mandatory)

Bug fix (change which fixes an issue)

How has this PR been tested? (mandatory)

Add a unit test to test the logic and there is no PessimisticLockNotFound error when sysbenching.

Does this PR affect documentation (docs) or release note? (mandatory)

No

Does this PR affect tidb-ansible update? (mandatory)

No

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

request's for_update_ts is greater than lock's for_update_ts Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

youjiali1995 · 2019-05-29T05:41:13Z

@coocood @zhangjinpeng1987 @AndreMouche PTAL

BusyJay · 2019-05-29T05:53:19Z

Horrible pull request title!

src/storage/mvcc/txn.rs

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

coocood · 2019-05-29T07:10:13Z

src/storage/mvcc/txn.rs

-                // Duplicated command. No need to overwrite the lock and data.
-                MVCC_DUPLICATE_CMD_COUNTER_VEC.prewrite.inc();
-                return Ok(());
+                // Abort on lock belonging to other transaction if


This block is too long, extract a function is better.

I think now it is clearer than extracting a function, and the function needs many parameters.

Only 4 parameters: self, lock, key, for_update_ts.

And also is_pessimistic_lock. The name of function is redundent, like check_pessimistic_prewrite_lock_conflict.

OK, then we can extract the lines after if is_pessimistic_lock block to a function.

The name can be check_non_pessimistic_lock_conflict

…prewrite-conflict Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

coocood · 2019-05-29T09:41:00Z

src/storage/mvcc/txn.rs

@@ -327,14 +336,15 @@ impl<S: Snapshot> MvccTxn<S> {
                key: key.into_raw()?,
            });
        }
+
        // No need to check data constraint, it's resolved by pessimistic locks.
        self.put_lock(


we can reduce 2 parameters for this function.

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

…iali1995/tikv into solve-pessimistic-prewrite-conflict

coocood · 2019-05-29T10:16:30Z

src/storage/mvcc/txn.rs

        value: Option<Value>,
-        is_pessimistic_txn: bool,
-        txn_size: u64,
+        options: &Options,
    ) {
        if value.is_none() || is_short_value(value.as_ref().unwrap()) {
            self.lock_key(


lock_key can reduce parameters too.

😅my fault.

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

…iali1995/tikv into solve-pessimistic-prewrite-conflict

coocood · 2019-05-29T10:26:31Z

LGTM

AndreMouche · 2019-05-30T07:04:36Z

src/storage/mod.rs

@@ -49,7 +49,7 @@ pub type Callback<T> = Box<dyn FnOnce(Result<T>) + Send>;
 // Short value max len must <= 255.
 pub const SHORT_VALUE_MAX_LEN: usize = 64;
 pub const SHORT_VALUE_PREFIX: u8 = b'v';
-pub const PESSIMISTIC_TXN: u8 = b'p';
+pub const FOR_UPDATE_TS_PREFIX: u8 = b'f';


I think it's not a good idea to change the prefix .. we will have two different kinds of prefix between versions.

Even if the prefix is the same, different versions won't be compatible because the old version doesn't record for_update_ts after the prefix.

I think it is ok since this feature is in developing.

AndreMouche · 2019-05-30T07:18:39Z

src/server/service/kv.rs

@@ -1322,6 +1322,7 @@ fn future_prewrite<E: Engine>(
    let mut options = Options::default();
    options.lock_ttl = req.get_lock_ttl();
    options.skip_constraint_check = req.get_skip_constraint_check();
+    options.for_update_ts = req.get_for_update_ts();


Why we need for_update_ts in prewrite, seems we already have it by acquire_pessimistic_lock?

By the way, why the for_update_ts is the same for all key in the prewrite command?

for_update_ts is used to resolve non-pessimistic-lock conflict.

TiDB uses the last for_update_ts as prewrite's.

AndreMouche · 2019-05-30T08:28:57Z

src/storage/mvcc/txn.rs

+        // is replaced atomicly.
+        //
+        // Optimistic lock's for_update_ts is its start_ts.
+        let lock_for_update_ts = if lock.for_update_ts > 0 {


When will it happen? Here we meet a pessimistic lock which does not belong to current transaction when we prewrite, which I think this shouldn't happen.

It is a non-pessimistic lock. The google doc has more info.

AndreMouche · 2019-05-30T08:31:13Z

src/storage/mvcc/txn.rs

+
+                self.check_non_pessimistic_lock_conflict(options.for_update_ts, &lock)?;
+            } else {
+                if lock.lock_type != LockType::Pessimistic {


How about using else if directly?

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

…iali1995/tikv into solve-pessimistic-prewrite-conflict

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

…iali1995/tikv into solve-pessimistic-prewrite-conflict

youjiali1995 · 2019-05-31T07:18:21Z

@coocood @AndreMouche @zhangjinpeng1987 PTAL

coocood · 2019-05-31T07:28:14Z

src/storage/mvcc/txn.rs

@@ -143,7 +143,7 @@ impl<S: Snapshot> MvccTxn<S> {

    // If the value is short, lock key and put value.
    // If not, lock key.
-    fn put_lock(
+    fn lock_key_and_put_value_if_short(


The logic is lock_key_and_put_value_if_long.
And I think the name is too complex.
How about just prewrite_key or prewrite_key_value?

Yes. I prefer prewrite_key_value

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

…iali1995/tikv into solve-pessimistic-prewrite-conflict

coocood · 2019-05-31T07:39:10Z

LGTM

youjiali1995 · 2019-06-04T05:17:21Z

@zhangjinpeng1987 @AndreMouche PTAL

zhangjinpeng87 · 2019-06-05T02:51:40Z

src/storage/mod.rs

                ..
            } => write!(
                f,
-                "kv::command::acquirepessimisticlock keys({}) @ {},{} | {:?}",
+                "kv::command::acquirepessimisticlock keys({}) @ {} | {:?}",


Better to display for_update_ts, it is useful for debugging.

src/storage/mvcc/txn.rs

zhangjinpeng87 · 2019-06-05T03:30:28Z

src/storage/mvcc/txn.rs

@@ -199,14 +178,40 @@ impl<S: Snapshot> MvccTxn<S> {
        Ok(())
    }

+    fn handle_non_pessimistic_lock_conflict(
+        &self,


Seems this function never use self.

It is a bit complex to call it without self.

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

zhangjinpeng87

LGTM

AndreMouche · 2019-06-06T04:28:20Z

src/storage/mvcc/txn.rs

+                primary: lock.primary,
+                ts: lock.ts,
+                // Set ttl to 0 so TiDB will resolve lock immediately.
+                ttl: 0,


what if this lock is a normal primary key, and the corresponding txn is not time out？

Signed-off-by: Yilin Chen <sticnarf@gmail.com> Makefile: make sure gdb is installed before sse4.2 check (tikv#4832) Signed-off-by: Kaige Ye <ye@kaige.org> Upgrade sys-info (tikv#4760) * *: upgrade sys-info crate This fixes a problem with the next toolchain upgrade where rust fails to link the native components of the crate. Signed-off-by: Brian Anderson <andersrb@gmail.com> * *: bump sys-info to 0.5.7 Signed-off-by: Brian Anderson <andersrb@gmail.com> Batch Top N Executor (tikv#4825) Signed-off-by: Breezewish <breezewish@pingcap.com> Add help message in doc:go-client-api.md (tikv#4763) * add help message in doc:go-client-api.md Signed-off-by: yy <cacheyy@qq.com> * update go-client-api.md Signed-off-by: yy <cacheyy@qq.com> Modify Makefile to distinguish between developer and packaging use cases (tikv#4687) * make: Add new "dist_release" rules To make the optimized build faster the existing "release" rules are going to changed such that they are not identical to the actual releases. Primarily they will not have debuginfo by default and will use thinLTO instead of LTO. This adds new "dist_release", etc rules for the CI/CD system to use. For now they are identical to the existing rules. After CI is updated the "release" rules will be changed. Signed-off-by: Brian Anderson <andersrb@gmail.com> * make: Document release rules Signed-off-by: Brian Anderson <andersrb@gmail.com> * Makefile: indicate use of fail_release Signed-off-by: Brian Anderson <andersrb@gmail.com> * Clarify the distinction in instruction set for release targets Signed-off-by: Brian Anderson <andersrb@gmail.com> Makefile: fix gdb check (tikv#4840) Signed-off-by: Kaige Ye <ye@kaige.org> pessimistic-txn: solve non-pessimistic-lock conflict (tikv#4801) * txn: replace is_pessimistic_lock to for_update_ts in Lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * pessimistic-txn: overwrite optimistic lock in pessimistic_prewrite if request's for_update_ts is greater than lock's for_update_ts Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * modify comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * add comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * return Error let TiDB to resolve lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> coprocessor: add batch aggregate function BitAnd/BitOr/BitXor (tikv#4824) Batch Top N Layered Benchmarks (tikv#4827) * Add Top N benchmarks Signed-off-by: Breezewish <breezewish@pingcap.com> * Address some comments in previous PRs Signed-off-by: Breezewish <breezewish@pingcap.com> coprocessor: add batch aggregate function Max/Min (tikv#4837) Implement RpnFunction MultiplyDecimal (tikv#4849) Signed-off-by: Breezewish <breezewish@pingcap.com> Add missing fsync calls in the snapshot module (tikv#4850) Signed-off-by: Ben Pig Chu <benpichu@gmail.com> use HTTP to enable jemalloc profile (tikv#4600) use HTTP to enable jemalloc profile Signed-off-by: Yang Keao <keao.yang@yahoo.com> coprocessor: use servo_arc in BatchTopNExecutor (tikv#4854) Signed-off-by: Yilin Chen <sticnarf@gmail.com> Fix clippy warnings Signed-off-by: Yilin Chen <sticnarf@gmail.com> Fix test Signed-off-by: Yilin Chen <sticnarf@gmail.com> Add docs to function.rs Signed-off-by: Yilin Chen <sticnarf@gmail.com> Add example output of the macro in the test of the macro. Signed-off-by: Yilin Chen <sticnarf@gmail.com> fix broken url for configuration options (tikv#4856) Signed-off-by: Yukang <moorekang@gmail.com> shrink the latch waiting list (tikv#4844) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Fix clippy Signed-off-by: Yilin Chen <sticnarf@gmail.com> scheduler use spin::Mutex (tikv#4829) * scheduler use spinlock Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Better panic info Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* txn: replace is_pessimistic_lock to for_update_ts in Lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * pessimistic-txn: overwrite optimistic lock in pessimistic_prewrite if request's for_update_ts is greater than lock's for_update_ts Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * modify comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * add comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * return Error let TiDB to resolve lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * address comment Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

youjiali1995 added 4 commits May 28, 2019 17:08

txn: replace is_pessimistic_lock to for_update_ts in Lock

ad2c0b7

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

pessimistic-txn: overwrite optimistic lock in pessimistic_prewrite if

928015e

request's for_update_ts is greater than lock's for_update_ts Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

modify comment

9dc44b4

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'master' into solve-pessimistic-prewrite-conflict

0c97e0a

youjiali1995 changed the title ~~pessimistic-txn: overwrite optimistic lock in pessimistic_prewrite if request's for_update_ts is greater than lock's for_update_ts~~ pessimistic-txn: solve optimistic lock conflict May 29, 2019

youjiali1995 added the sig/transaction SIG: Transaction label May 29, 2019

youjiali1995 requested review from zhangjinpeng87 and AndreMouche May 29, 2019 06:33

coocood reviewed May 29, 2019

View reviewed changes

src/storage/mvcc/txn.rs Outdated Show resolved Hide resolved

coocood reviewed May 29, 2019

View reviewed changes

src/storage/mvcc/txn.rs Outdated Show resolved Hide resolved

coocood reviewed May 29, 2019

View reviewed changes

src/storage/mvcc/txn.rs Outdated Show resolved Hide resolved

address comment

5881075

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

coocood reviewed May 29, 2019

View reviewed changes

youjiali1995 added 2 commits May 29, 2019 17:08

Merge branch 'master' of github.com:tikv/tikv into solve-pessimistic-…

0a88057

…prewrite-conflict Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'master' into solve-pessimistic-prewrite-conflict

8ffe875

coocood reviewed May 29, 2019

View reviewed changes

address comment

e5895b6

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

youjiali1995 changed the title ~~pessimistic-txn: solve optimistic lock conflict~~ pessimistic-txn: solve non-pessimistic-lock conflict May 29, 2019

youjiali1995 added 3 commits May 29, 2019 18:10

address comment

b7d935c

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'solve-pessimistic-prewrite-conflict' of github.com:youj…

5fef132

…iali1995/tikv into solve-pessimistic-prewrite-conflict

Merge branch 'master' into solve-pessimistic-prewrite-conflict

bf054bc

coocood reviewed May 29, 2019

View reviewed changes

youjiali1995 added 3 commits May 29, 2019 18:22

address comment

cc0490a

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'solve-pessimistic-prewrite-conflict' of github.com:youj…

9a242fc

…iali1995/tikv into solve-pessimistic-prewrite-conflict

Merge branch 'master' into solve-pessimistic-prewrite-conflict

cdd9a21

AndreMouche reviewed May 30, 2019

View reviewed changes

add comment

1666be9

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

youjiali1995 added 2 commits May 30, 2019 17:33

Merge branch 'solve-pessimistic-prewrite-conflict' of github.com:youj…

1d4b0ff

…iali1995/tikv into solve-pessimistic-prewrite-conflict

Merge branch 'master' into solve-pessimistic-prewrite-conflict

4d5e7fb

youjiali1995 added the S: DNM label May 30, 2019

youjiali1995 added 2 commits May 30, 2019 20:28

return Error let TiDB to resolve lock

7d2f655

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'solve-pessimistic-prewrite-conflict' of github.com:youj…

14f6be9

…iali1995/tikv into solve-pessimistic-prewrite-conflict

youjiali1995 removed the S: DNM label May 31, 2019

Merge branch 'master' into solve-pessimistic-prewrite-conflict

a3e9dd5

coocood reviewed May 31, 2019

View reviewed changes

youjiali1995 added 2 commits May 31, 2019 15:36

address comment

098241a

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'solve-pessimistic-prewrite-conflict' of github.com:youj…

ac1b250

…iali1995/tikv into solve-pessimistic-prewrite-conflict

Merge branch 'master' into solve-pessimistic-prewrite-conflict

f28cac1

zhangjinpeng87 reviewed Jun 5, 2019

View reviewed changes

src/storage/mvcc/txn.rs Show resolved Hide resolved

zhangjinpeng87 reviewed Jun 5, 2019

View reviewed changes

youjiali1995 added 2 commits June 5, 2019 11:47

address comment

b095cc6

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Merge branch 'master' into solve-pessimistic-prewrite-conflict

e3459f0

zhangjinpeng87 approved these changes Jun 5, 2019

View reviewed changes

youjiali1995 added the status/LGT1 Status: PR - There is already 1 approval label Jun 6, 2019

AndreMouche reviewed Jun 6, 2019

View reviewed changes

AndreMouche approved these changes Jun 6, 2019

View reviewed changes

Merge branch 'master' into solve-pessimistic-prewrite-conflict

b7c1a74

youjiali1995 merged commit ae73f6c into tikv:master Jun 6, 2019

youjiali1995 deleted the solve-pessimistic-prewrite-conflict branch June 6, 2019 05:45

pessimistic-txn: solve non-pessimistic-lock conflict #4801

pessimistic-txn: solve non-pessimistic-lock conflict #4801

Conversation

youjiali1995 commented May 29, 2019

What have you changed? (mandatory)

What are the type of the changes? (mandatory)

How has this PR been tested? (mandatory)

Does this PR affect documentation (docs) or release note? (mandatory)

Does this PR affect tidb-ansible update? (mandatory)

youjiali1995 commented May 29, 2019

BusyJay commented May 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood May 29, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood commented May 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

youjiali1995 commented May 31, 2019

coocood May 31, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood commented May 31, 2019

youjiali1995 commented Jun 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangjinpeng87 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coocood May 29, 2019 •

edited

coocood May 31, 2019 •

edited