Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jepsen monotonic test failed with anomalies #48802

Closed
zyguan opened this issue Nov 22, 2023 · 1 comment · Fixed by tikv/tikv#16060
Closed

Jepsen monotonic test failed with anomalies #48802

zyguan opened this issue Nov 22, 2023 · 1 comment · Fixed by tikv/tikv#16060
Labels
affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.0 affects-6.1 affects-6.2 affects-6.3 affects-6.4 affects-6.5 affects-6.6 affects-7.0 affects-7.1 affects-7.2 affects-7.3 affects-7.4 affects-7.5 severity/critical sig/transaction SIG:Transaction type/bug This issue is a bug.

Comments

@zyguan
Copy link
Contributor

zyguan commented Nov 22, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

run jepsen test

2. What did you expect to see? (Required)

no error reported

3. What did you see instead (Required)

INFO [2023-11-21 07:34:08,613] jepsen test runner - jepsen.core {:perf
 {:latency-graph {:valid? true},
  :rate-graph {:valid? true},
  :valid? true},
 :workload
 {:cycle
  {:valid? false,
   :scc-count 1,
   :cycles
   ["Let:\n  T1 = {:type :ok, :f :read, :value {0 1889, 1 1875, 2 1892, 3 1814, 4 1794, 5 1912, 6 1909, 7 1719}, :process 69, :time 241358622457, :txn-info {:start_ts 445789485148667916}, :index 59501}\n  T2 = {:type :ok, :f :read, :value {0 1889, 1 1875, 2 1893, 3 1813, 4 1794, 5 1912, 6 1909, 7 1719}, :process 111, :time 241400793624, :txn-info {:start_ts 445789485135560758}, :index 59523}\n\nThen:\n  - T1 < T2, because T1 observed 2 = 1892, and T2 observed a higher value 1893.\n  - However, T2 < T1, because T2 observed 3 = 1813, and T1 observed a higher value 1814: a contradiction!"]},
  :timeline {:valid? true},
  :valid? false},
 :valid? false}


Analysis invalid! (ノಥ益ಥ)ノ ┻━┻

4. What is your TiDB version? (Required)

master & release-7.5

@zyguan zyguan added the type/bug This issue is a bug. label Nov 22, 2023
@ti-chi-bot ti-chi-bot bot added may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Nov 22, 2023
@overvenus overvenus added affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-6.0 affects-6.1 affects-6.2 affects-6.3 affects-6.4 affects-6.5 affects-6.6 affects-7.0 affects-7.1 affects-7.2 and removed may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Nov 23, 2023
@cfzjywxk
Copy link
Contributor

Close this and create a detailed issue in the tikv repo tikv/tikv#16069.

ti-chi-bot bot added a commit to tikv/tikv that referenced this issue Nov 27, 2023
close #16069, close pingcap/tidb#48802

When a peer is removed, it is necessary to update its apply state because
this peer may be simultaneously taking a snapshot. An outdated apply state
will invalidate the coprocessor cache assumption and potentially lead to
a violation of linearizability (returning stale cache).

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 27, 2023
close tikv#16069, close pingcap/tidb#48802

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 27, 2023
close tikv#16069, close pingcap/tidb#48802

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 27, 2023
close tikv#16069, close pingcap/tidb#48802

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit to tikv/tikv that referenced this issue Nov 27, 2023
close #16069, close pingcap/tidb#48802

When a peer is removed, it is necessary to update its apply state because
this peer may be simultaneously taking a snapshot. An outdated apply state
will invalidate the coprocessor cache assumption and potentially lead to
a violation of linearizability (returning stale cache).

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
overvenus added a commit to ti-chi-bot/tikv that referenced this issue Nov 27, 2023
close tikv#16069, close pingcap/tidb#48802

When a peer is removed, it is necessary to update its apply state because
this peer may be simultaneously taking a snapshot. An outdated apply state
will invalidate the coprocessor cache assumption and potentially lead to
a violation of linearizability (returning stale cache).

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
ti-chi-bot bot pushed a commit to tikv/tikv that referenced this issue Nov 27, 2023
close #16069, close pingcap/tidb#48802

When a peer is removed, it is necessary to update its apply state because
this peer may be simultaneously taking a snapshot. An outdated apply state
will invalidate the coprocessor cache assumption and potentially lead to
a violation of linearizability (returning stale cache).

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
ti-chi-bot bot added a commit to tikv/tikv that referenced this issue Nov 29, 2023
close #16069, close pingcap/tidb#48802

When a peer is removed, it is necessary to update its apply state because
this peer may be simultaneously taking a snapshot. An outdated apply state
will invalidate the coprocessor cache assumption and potentially lead to
a violation of linearizability (returning stale cache).

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@jebter jebter added the sig/transaction SIG:Transaction label May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.0 affects-6.1 affects-6.2 affects-6.3 affects-6.4 affects-6.5 affects-6.6 affects-7.0 affects-7.1 affects-7.2 affects-7.3 affects-7.4 affects-7.5 severity/critical sig/transaction SIG:Transaction type/bug This issue is a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants