txn: Support check for_update_ts when prewriting #14492

MyonKeminta · 2023-03-30T10:39:18Z

What is changed and how it works?

Issue Number: Ref #14311
Requires: pingcap/kvproto#1096

What's Changed:

Supports checking for_update_ts for specific keys during prewrite to avoid potential lost update that might be caused by allowing locking with conflict.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

(TODO)

Unit test
Integration test

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

None

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot · 2023-03-30T10:39:19Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

cfzjywxk
ekexium

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…yonKeminta/tikv into m/prewrite-check-for-update-ts

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…her PR Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot · 2023-04-11T14:12:14Z

This pull request has been accepted and is ready to merge.

Commit hash: e85a9ad

MyonKeminta · 2023-04-12T06:04:44Z

/test

MyonKeminta · 2023-04-12T06:28:03Z

/test

MyonKeminta · 2023-04-12T06:48:25Z

????
/test

MyonKeminta · 2023-04-12T07:13:11Z

Why adding the need-cherry-pick-release-7.0 tag while 7.0 is a DMR version that rejects any more updates?

MyonKeminta · 2023-04-12T07:36:34Z

[2023-04-12T07:29:51.450Z] thread 'test_merge::test_merge' panicked at 'assertion failed: `(left == right)`
[2023-04-12T07:29:51.450Z]   left: `[107, 52]`,
[2023-04-12T07:29:51.450Z]  right: `[107, 51]`', components/raftstore-v2/tests/integrations/cluster.rs:869:17
[2023-04-12T07:29:51.450Z] stack backtrace:
[2023-04-12T07:29:51.450Z]    0: rust_begin_unwind
[2023-04-12T07:29:51.450Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
[2023-04-12T07:29:51.450Z]    1: core::panicking::panic_fmt
[2023-04-12T07:29:51.450Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
[2023-04-12T07:29:51.450Z]    2: core::panicking::assert_failed_inner
[2023-04-12T07:29:51.450Z]    3: core::panicking::assert_failed
[2023-04-12T07:29:51.450Z]    4: raftstore_v2_integrations::cluster::merge_helper::merge_region
[2023-04-12T07:29:51.450Z]    5: raftstore_v2_integrations::test_merge::test_merge
[2023-04-12T07:29:51.450Z]    6: raftstore_v2_integrations::test_merge::test_merge::{{closure}}
[2023-04-12T07:29:51.450Z]    7: core::ops::function::FnOnce::call_once
[2023-04-12T07:29:51.450Z]    8: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}
[2023-04-12T07:29:51.450Z]    9: core::ops::function::FnOnce::call_once{{vtable.shim}}
[2023-04-12T07:29:51.450Z]   10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[2023-04-12T07:29:51.450Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
[2023-04-12T07:29:51.450Z] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2023-04-12T07:29:51.450Z] 
[2023-04-12T07:29:51.450Z]    TRY 3 TMT [ 120.037s] raftstore-v2::raftstore-v2-integrations test_transfer_leader::test_transfer_leader
[2023-04-12T07:29:51.450Z] 
[2023-04-12T07:29:51.450Z] --- TRY 3 STDOUT:        raftstore-v2::raftstore-v2-integrations test_transfer_leader::test_transfer_leader ---

[2023-04-12T07:29:51.451Z] thread 'cases::test_pd_client::test_retry' panicked at 'called `Result::unwrap()` on an `Err` value: Other("[components/pd_client/src/client_v2.rs:312]: Connection unavailable GRPC_CHANNEL_TRANSIENT_FAILURE")', tests/failpoints/cases/test_pd_client.rs:147:42
[2023-04-12T07:29:51.451Z] stack backtrace:
[2023-04-12T07:29:51.451Z]    0: rust_begin_unwind
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
[2023-04-12T07:29:51.451Z]    1: core::panicking::panic_fmt
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
[2023-04-12T07:29:51.451Z]    2: core::result::unwrap_failed
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/result.rs:1791:5
[2023-04-12T07:29:51.451Z]    3: core::result::Result<T,E>::unwrap
[2023-04-12T07:29:51.451Z]    4: failpoints::cases::test_pd_client::run_on_bad_connection
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:147:9
[2023-04-12T07:29:51.451Z]    5: failpoints::cases::test_pd_client::test_retry::test_retry_success
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:201:13
[2023-04-12T07:29:51.451Z]    6: failpoints::cases::test_pd_client::test_retry
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:212:5
[2023-04-12T07:29:51.451Z]    7: failpoints::cases::test_pd_client::test_retry::{{closure}}
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:186:17
[2023-04-12T07:29:51.451Z]    8: core::ops::function::FnOnce::call_once
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
[2023-04-12T07:29:51.451Z]    9: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}
[2023-04-12T07:29:51.451Z]   10: core::ops::function::FnOnce::call_once{{vtable.shim}}
[2023-04-12T07:29:51.451Z]   11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9

Are these tests always as unstable as this? I don't think this PR has anything to do with them

MyonKeminta · 2023-04-12T07:47:29Z

/test

MyonKeminta · 2023-04-12T08:10:13Z

/test

MyonKeminta · 2023-04-12T08:54:15Z

/test

MyonKeminta · 2023-04-12T09:21:53Z

/test

MyonKeminta · 2023-04-12T09:33:15Z

/test

MyonKeminta · 2023-04-12T09:53:44Z

/test

MyonKeminta · 2023-04-12T09:59:09Z

[2023-04-12T07:29:51.450Z] thread 'test_merge::test_merge' panicked at 'assertion failed: `(left == right)`
[2023-04-12T07:29:51.450Z]   left: `[107, 52]`,
[2023-04-12T07:29:51.450Z]  right: `[107, 51]`', components/raftstore-v2/tests/integrations/cluster.rs:869:17
[2023-04-12T07:29:51.450Z] stack backtrace:
[2023-04-12T07:29:51.450Z]    0: rust_begin_unwind
[2023-04-12T07:29:51.450Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
[2023-04-12T07:29:51.450Z]    1: core::panicking::panic_fmt
[2023-04-12T07:29:51.450Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
[2023-04-12T07:29:51.450Z]    2: core::panicking::assert_failed_inner
[2023-04-12T07:29:51.450Z]    3: core::panicking::assert_failed
[2023-04-12T07:29:51.450Z]    4: raftstore_v2_integrations::cluster::merge_helper::merge_region
[2023-04-12T07:29:51.450Z]    5: raftstore_v2_integrations::test_merge::test_merge
[2023-04-12T07:29:51.450Z]    6: raftstore_v2_integrations::test_merge::test_merge::{{closure}}
[2023-04-12T07:29:51.450Z]    7: core::ops::function::FnOnce::call_once
[2023-04-12T07:29:51.450Z]    8: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}
[2023-04-12T07:29:51.450Z]    9: core::ops::function::FnOnce::call_once{{vtable.shim}}
[2023-04-12T07:29:51.450Z]   10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[2023-04-12T07:29:51.450Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
[2023-04-12T07:29:51.450Z] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2023-04-12T07:29:51.450Z] 
[2023-04-12T07:29:51.450Z]    TRY 3 TMT [ 120.037s] raftstore-v2::raftstore-v2-integrations test_transfer_leader::test_transfer_leader
[2023-04-12T07:29:51.450Z] 
[2023-04-12T07:29:51.450Z] --- TRY 3 STDOUT:        raftstore-v2::raftstore-v2-integrations test_transfer_leader::test_transfer_leader ---

[2023-04-12T07:29:51.451Z] thread 'cases::test_pd_client::test_retry' panicked at 'called `Result::unwrap()` on an `Err` value: Other("[components/pd_client/src/client_v2.rs:312]: Connection unavailable GRPC_CHANNEL_TRANSIENT_FAILURE")', tests/failpoints/cases/test_pd_client.rs:147:42
[2023-04-12T07:29:51.451Z] stack backtrace:
[2023-04-12T07:29:51.451Z]    0: rust_begin_unwind
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
[2023-04-12T07:29:51.451Z]    1: core::panicking::panic_fmt
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
[2023-04-12T07:29:51.451Z]    2: core::result::unwrap_failed
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/result.rs:1791:5
[2023-04-12T07:29:51.451Z]    3: core::result::Result<T,E>::unwrap
[2023-04-12T07:29:51.451Z]    4: failpoints::cases::test_pd_client::run_on_bad_connection
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:147:9
[2023-04-12T07:29:51.451Z]    5: failpoints::cases::test_pd_client::test_retry::test_retry_success
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:201:13
[2023-04-12T07:29:51.451Z]    6: failpoints::cases::test_pd_client::test_retry
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:212:5
[2023-04-12T07:29:51.451Z]    7: failpoints::cases::test_pd_client::test_retry::{{closure}}
[2023-04-12T07:29:51.451Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_pd_client.rs:186:17
[2023-04-12T07:29:51.451Z]    8: core::ops::function::FnOnce::call_once
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
[2023-04-12T07:29:51.451Z]    9: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}
[2023-04-12T07:29:51.451Z]   10: core::ops::function::FnOnce::call_once{{vtable.shim}}
[2023-04-12T07:29:51.451Z]   11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
[2023-04-12T07:29:51.451Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9

Are these tests always as unstable as this? I don't think this PR has anything to do with them

cc @tabokie @BusyJay Do you have any idea?

MyonKeminta · 2023-04-12T10:06:06Z

/test

MyonKeminta · 2023-04-12T10:30:23Z

/test

cfzjywxk · 2023-04-12T10:56:24Z

/test

BusyJay · 2023-04-12T11:50:30Z

No, it's not expected.

MyonKeminta added 2 commits March 30, 2023 14:30

update kvproto

243c131

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

txn: Support check for_update_ts when prewriting

036916f

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot added release-note-none needs-cherry-pick-release-7.0 do-not-merge/work-in-progress size/L labels Mar 30, 2023

Fix build

c1b3367

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta mentioned this pull request Mar 30, 2023

Support specifying checking forUpdateTS in prewrite tikv/client-go#756

Merged

MyonKeminta added 2 commits March 31, 2023 20:28

Add unit test to action/prewrite

6925586

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Merge branch 'm/prewrite-check-for-update-ts' of https://github.com/M…

53ab086

…yonKeminta/tikv into m/prewrite-check-for-update-ts

ti-chi-bot added size/XL and removed size/L labels Mar 31, 2023

Add more tests

a273936

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot added size/XXL and removed size/XL labels Apr 4, 2023

MyonKeminta added 3 commits April 4, 2023 14:44

Exclude the case that lock.for_update_ts == min_commit_ts

1e03cfc

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

fix tests and typos

c301221

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Fix build

97fdd06

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta force-pushed the m/prewrite-check-for-update-ts branch from 65d8386 to 97fdd06 Compare April 4, 2023 07:05

MyonKeminta added 7 commits April 4, 2023 15:30

Fix build

e8f140f

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Fix tests

8dea574

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Fix lint

95c4ab0

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Remove the code about async commit safety check as it will be in anot…

63df17d

…her PR Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Update kvproto

ce549c3

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Disallow lock.for_update_ts < expected

19ab2e5

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Fix build

da7d3d1

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

MyonKeminta marked this pull request as ready for review April 6, 2023 13:44

ti-chi-bot removed the do-not-merge/work-in-progress label Apr 6, 2023

Fix build

ea37a82

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot added the status/can-merge Status: Can merge to base branch label Apr 11, 2023

Merge branch 'master' into m/prewrite-check-for-update-ts

b1a098f

ti-chi-bot added the needs-cherry-pick-release-7.0 label Apr 11, 2023

MyonKeminta removed the needs-cherry-pick-release-7.0 label Apr 12, 2023

Merge branch 'master' into m/prewrite-check-for-update-ts

20c1739

ti-chi-bot added the needs-cherry-pick-release-7.0 label Apr 12, 2023

MyonKeminta removed the needs-cherry-pick-release-7.0 label Apr 12, 2023

Merge branch 'master' into m/prewrite-check-for-update-ts

01ee044

ti-chi-bot added the needs-cherry-pick-release-7.0 label Apr 12, 2023

MyonKeminta removed the needs-cherry-pick-release-7.0 label Apr 12, 2023

ti-chi-bot merged commit 3630ba9 into tikv:master Apr 12, 2023
7 checks passed

ti-chi-bot added this to the Pool milestone Apr 12, 2023

MyonKeminta deleted the m/prewrite-check-for-update-ts branch April 12, 2023 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

txn: Support check for_update_ts when prewriting #14492

txn: Support check for_update_ts when prewriting #14492

MyonKeminta commented Mar 30, 2023 •

edited

ti-chi-bot commented Mar 30, 2023 •

edited

ti-chi-bot commented Apr 11, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

cfzjywxk commented Apr 12, 2023

BusyJay commented Apr 12, 2023

txn: Support check for_update_ts when prewriting #14492

txn: Support check for_update_ts when prewriting #14492

Conversation

MyonKeminta commented Mar 30, 2023 • edited

What is changed and how it works?

Related changes

Check List

Release note

ti-chi-bot commented Mar 30, 2023 • edited

ti-chi-bot commented Apr 11, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

MyonKeminta commented Apr 12, 2023

cfzjywxk commented Apr 12, 2023

BusyJay commented Apr 12, 2023

MyonKeminta commented Mar 30, 2023 •

edited

ti-chi-bot commented Mar 30, 2023 •

edited