Async commit does not ensure linearizability

[sticnarf]

Bug Report

Read in the same session

Read with max u64 immediately after an async commit transaction. Because the lock in the previous transaction may be not committed yet, and locks can be ignored if we read with max u64. So we may not read the latest changes from the last async commit transaction.

Causal consistency across nodes

Example:

At first the values of k1 and k2 are all 0, they can locate in different regions.

We commit k1=1@100 at first. The max_read_ts of the k2 region leader is 50.

Then, we prewrite and commit k2=1@51 using async commit.

Now, a transaction with start_ts=80, reads k1 and k2 in the same transaction. We get k1=0, k2=1.

We break an external consistency that k2 commits later than k1.

Affected version

master

[sticnarf]

For the first problem, we can change the check_ts_conflict function. If the lock is from an async commit transaction, we cannot ignore it even if the read ts is max u64.

[youjiali1995]

We commit k1=1@100 at first. The max_read_ts of the k2 region leader is 50.
Then, we prewrite and commit k2=1@51 using async commit.

It's impossible if transactions modify k1 and k2 are not concurrent. In your case, all transactions are concurrent so the order is OK.

I can't think of a case that async commit in TiDB can break external consistency. And because we get start ts from PD which can avoid causal reverse.

[sticnarf]

@youjiali1995
A more concrete example:

1. An external client starts two sessions A and B.
2. Session A starts T1, start_ts = 1.
3. Session B starts T2, start_ts = 2.
4. Client starts session C, it starts a transaction T3, start_ts = 80.
5. Session A prewrites k1=1.
6. Session A commits T1 with commit_ts = 100.
7. Session B prewrites k2=1 using async commit, min_commit_ts = 51.
8. Session B commits T2 with commit_ts = 51.
9. Session C reads k1 and k2.

[sticnarf]

@youjiali1995
A more concrete example:

1. An external client starts two sessions A and B.

2. Session A starts T1, start_ts = 1.

3. Session B starts T2, start_ts = 2.

4. Client starts session C, it starts a transaction T3, start_ts = 80.

5. Session A prewrites k1=1.

6. Session A commits T1 with commit_ts = 100.

7. Session B prewrites k2=1 using async commit, min_commit_ts = 51.

8. Session B commits T2 with commit_ts = 51.

9. Session C reads k1 and k2.

Session B triggers commit after it receives the success from session A (between 6 and 7)

[MyonKeminta]

I'm actually not very sure if we need to guarantee this. @coocood PTAL

[jackysp]

Can Jepsen test find this out?

[youjiali1995]

I see. It's another kind of causal reverse. We can't avoid it but I think it's not very severe because it rarely happens. We can implement causality tokens to solve it which is passed between transactions if needed.

[sticnarf]

It's quite similar to the causal reverse crdb mentioned in their blog. So I think it can be detected by Jepsen too. @jackysp

[youjiali1995]

If a user depends on external consistency, we can disable async commit. Actually, we can't offer the true external consistency even with the timestamp oracle which requires serializability isolation.

[coocood]

This anomaly requires one client maintains two transactions at the same time, are there any existing application do it this way?

[sticnarf]

This anomaly requires one client maintains two transactions at the same time, are there any existing application do it this way?

Usually operations with causality are done in a single transaction. But maybe one client receives success and then inform another client to commit? I don't know...

But this is indeed a change in consistency guarantee which should be mentioned and make users aware of.

[coocood]

I can think of an example, an application A call service B inside a transaction, and service B do another transaction to get a value, then application A use that value to commit its transaction.

But this case application A and service B are separated components, the inconsistency should not matter much.