Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default auto-retry mechanism causes TiDB to lose updates by default #10075

Closed
aphyr opened this issue Apr 8, 2019 · 2 comments
Closed

Default auto-retry mechanism causes TiDB to lose updates by default #10075

aphyr opened this issue Apr 8, 2019 · 2 comments
Labels

Comments

@aphyr
Copy link

@aphyr aphyr commented Apr 8, 2019

The auto-retry mechanism, enabled by default in TiDB 2.1.7, retries transaction failures by blindly re-applying writes from the failed transaction in a fresh transaction. This effectively bypasses the conflict detection required by snapshot isolation: when a conflict is encountered, transactions should abort, but instead, they can go on to commit their conflicting writes regardless of conflicts. The result is lost updates.

There are certain classes of transactions which can be safely retried--for instance, blind writes, in-place updates, and transactions in which no client-visible reads occur before a write. However, TiDB assumes all transactions can be safely retried, which violates TiDB's claim of snapshot isolation.

I suggest disabling the retry mechanism by default.

  1. What did you do?

With Jepsen d003233c0e7e9f6d9f841647f661a868cd0a222a, run

lein run test --time-limit 60 --concurrency 2n -w bank --auto-retry

This test transfers money between simulated bank accounts; under snapshot isolation, the total of all accounts should remain constant. However, with automatic retries, under normal operating conditions without failures, the total balance drifts over time: transactions observe values in transaction T1, and, thanks to retries, commit new values based on those from T1 in a fresh transaction T2.

  1. What did you expect to see?

The total of all accounts should be constant.

  1. What did you see instead?

The total of all accounts fluctuates over time: http://jepsen.io.s3.amazonaws.com/analyses/tidb-2.1.7/20190408T132351-auto-retry.zip.

bank (7)

  1. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

v2.1.7-linux-amd64

@jackysp
Copy link
Member

@jackysp jackysp commented May 17, 2019

Hi @aphyr , the PR #10339 which disabled the auto retry by default has already been merged.

@aphyr
Copy link
Author

@aphyr aphyr commented May 29, 2019

Fixed in 3.0.0-rc.2!

@aphyr aphyr closed this May 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants