diff --git a/TOC.md b/TOC.md index 87f207a46cd6a..f71e30abc05e2 100644 --- a/TOC.md +++ b/TOC.md @@ -264,8 +264,8 @@ - [Certificate-Based Authentication](/reference/security/cert-based-authentication.md) + Transactions - [Overview](/reference/transactions/overview.md) - - [Transaction Model](/reference/transactions/transaction-model.md) - [Isolation Levels](/reference/transactions/transaction-isolation.md) + - [Optimistic Transactions](/reference/transactions/transaction-optimistic.md) - [Pessimistic Transactions](/reference/transactions/transaction-pessimistic.md) + System Databases - [`mysql`](/reference/system-databases/mysql.md) diff --git a/glossary.md b/glossary.md index dd1d60b829501..62546d703e283 100644 --- a/glossary.md +++ b/glossary.md @@ -6,6 +6,20 @@ category: glossary # Glossary +## A + +### ACID + +ACID refers to the four key properties of a transaction: atomicity, consistency, isolation, and durability. Each of these properties is described below. + +- **Atomicity** means that either all the changes of an operation are performed, or none of them are. TiDB ensures the atomicity of the [Region](#region) that stores the Primary Key to achieve the atomicity of transactions. + +- **Consistency** means that transactions always bring the database from one consistent state to another. In TiDB, data consistency is ensured before writing data to the memory. + +- **Isolation** means that a transaction in process is invisible to other transactions until it completes. This allows concurrent transactions to read and write data without sacrificing consistency. TiDB currently supports the isolation level of `REPEATABLE READ`. + +- **Durability** means that once a transaction is committed, it remains committed even in the event of a system failure. TiKV uses persistent storage to ensure durability. + ## L ### leader/follower/learner @@ -64,4 +78,4 @@ Schedulers are components in PD that generate scheduling tasks. Each scheduler i ### Store -A store refers to the storage node in the TiKV cluster (an instance of `tikv-server`). Each store has a corresponding TiKV instance. \ No newline at end of file +A store refers to the storage node in the TiKV cluster (an instance of `tikv-server`). Each store has a corresponding TiKV instance. diff --git a/key-features.md b/key-features.md index db3fe4111aaf0..713f64559b69c 100644 --- a/key-features.md +++ b/key-features.md @@ -26,7 +26,7 @@ We believe that being able to replicate in both directions lowers the risk when TiDB internally shards table into small range-based chunks that we refer to as "Regions". Each Region defaults to approximately 100MiB in size, and TiDB uses a Two-phase commit internally to ensure that Regions are maintained in a transactionally consistent way. -Transactions in TiDB are strongly consistent, with snapshot isolation level consistency. For more information, see transaction [behavior and performance differences](/reference/transactions/transaction-model.md). This makes TiDB more comparable to traditional relational databases in semantics than some of the newer NoSQL systems using eventual consistency. +Transactions in TiDB are strongly consistent, with snapshot isolation level consistency. For more information, see transaction [behavior and performance differences](/reference/transactions/transaction-isolation.md). This makes TiDB more comparable to traditional relational databases in semantics than some of the newer NoSQL systems using eventual consistency. These behaviors are transparent to your application(s), which only need to connect to TiDB using a MySQL 5.7 compatible client library. diff --git a/media/2pc-in-tidb.png b/media/2pc-in-tidb.png new file mode 100644 index 0000000000000..2e066cf25e66e Binary files /dev/null and b/media/2pc-in-tidb.png differ diff --git a/media/optimistic-transaction-metric.png b/media/optimistic-transaction-metric.png new file mode 100644 index 0000000000000..dd7373e6ffa89 Binary files /dev/null and b/media/optimistic-transaction-metric.png differ diff --git a/reference/configuration/tidb-server/tidb-specific-variables.md b/reference/configuration/tidb-server/tidb-specific-variables.md index 3790e35c95aa0..4310091cf2d0d 100644 --- a/reference/configuration/tidb-server/tidb-specific-variables.md +++ b/reference/configuration/tidb-server/tidb-specific-variables.md @@ -250,19 +250,19 @@ set @@global.tidb_distsql_scan_concurrency = 10 - Scope: SESSION | GLOBAL - Default value: 10 -- When a transaction encounters retryable errors (such as transaction conflicts, over slow transaction commit, or table schema changes), this transaction can be re-executed. This variable is used to set the maximum number of the retries. +- This variable is used to set the maximum number of the retries. When a transaction encounters retryable errors (such as transaction conflicts, very slow transaction commit, or table schema changes), this transaction is re-executed according to this variable. Note that setting `tidb_retry_limit` to `0` disables the automatic retry. ### tidb_disable_txn_auto_retry - Scope: SESSION | GLOBAL - Default: on -- This variable is used to set whether to disable automatic retry of explicit transactions. The default value of `on` means that transactions will not automatically retry in TiDB and `COMMIT` statements might return errors that need to be handled in the application layer. +- This variable is used to set whether to disable the automatic retry of explicit transactions. The default value of `on` means that transactions will not automatically retry in TiDB and `COMMIT` statements might return errors that need to be handled in the application layer. Setting the value to `off` means that TiDB will automatically retry transactions, resulting in fewer errors from `COMMIT` statements. Be careful when making this change, because it might result in lost updates. This variable does not affect automatically committed implicit transactions and internally executed transactions in TiDB. The maximum retry count of these transactions is determined by the value of `tidb_retry_limit`. - To decide whether you can enable automatic retry, see [automatic retry and anomalies caused by automatic retry](/reference/transactions/transaction-isolation.md#automatic-retry-and-transactional-anomalies-caused-by-automatic-retry). + For more details, see [limits of retry](/reference/transactions/transaction-optimistic.md#limits-of-retry). ### tidb_backoff_weight diff --git a/reference/mysql-compatibility.md b/reference/mysql-compatibility.md index 31803813ad212..d0ce524627ec1 100644 --- a/reference/mysql-compatibility.md +++ b/reference/mysql-compatibility.md @@ -14,7 +14,7 @@ However, TiDB does not support some of MySQL features or behaves differently fro > **Note:** > -> This page refers to general differences between MySQL and TiDB. Please also see the dedicated pages for [Security](/reference/security/compatibility.md) and [Transaction Model](/reference/transactions/transaction-model.md) compatibility. +> This page refers to general differences between MySQL and TiDB. Refer to the dedicated pages for [Security](/reference/security/compatibility.md) and [Pessimistic Transaction Model](/reference/transactions/transaction-pessimistic.md#difference-with-mysql-innodb) compatibility. ## Unsupported features diff --git a/reference/sql/statements/load-data.md b/reference/sql/statements/load-data.md index 9eee7917502a3..cb687f83274a5 100644 --- a/reference/sql/statements/load-data.md +++ b/reference/sql/statements/load-data.md @@ -58,5 +58,5 @@ In the above example, `x'2c'` is the hexadecimal representation of the `,` chara ## See also * [INSERT](/reference/sql/statements/insert.md) -* [Transaction Model](/reference/transactions/transaction-model.md) +* [Optimistic Transaction Model](/reference/transactions/transaction-optimistic.md) * [Import Example Database](/how-to/get-started/import-example-database.md) diff --git a/reference/sql/statements/select.md b/reference/sql/statements/select.md index 03f63e59eaa67..ce6eaf5fc4dd2 100644 --- a/reference/sql/statements/select.md +++ b/reference/sql/statements/select.md @@ -74,8 +74,8 @@ The `SELECT` statement is used to read data from TiDB. |`HAVING where_condition` | The `HAVING` clause and the `WHERE` clause are both used to filter the results. The `HAVING` clause filters the results of `GROUP BY`, while the `WHERE` clause filter the results before aggregation. | |`ORDER BY` | The `ORDER BY` clause is used to sort the data in ascending or descending order, based on columns, expressions or items in the `select_expr` list.| |`LIMIT` | The `LIMIT` clause can be used to constrain the number of rows. `LIMIT` takes one or two numeric arguments. With one argument, the argument specifies the maximum number of rows to return, the first row to return is the first row of the table by default; with two arguments, the first argument specifies the offset of the first row to return, and the second specifies the maximum number of rows to return.| -| `FOR UPDATE` | The `SELECT FOR UPDATE` clause locks all the data in the result sets to detect concurrent updates from other transactions. Data that match the query conditions but do not exist in the result sets are not read-locked, such as the row data written by other transactions after the current transaction is started. TiDB uses the [Optimistic Transaction Model](/reference/transactions/transaction-model.md). The transaction conflicts are not detected in the statement execution phase. Therefore, the current transaction does not block other transactions from executing `UPDATE`, `DELETE` or `SELECT FOR UPDATE` like other databases such as PostgreSQL. In the committing phase, the rows read by `SELECT FOR UPDATE` are committed in two phases, which means they can also join the conflict detection. If write conflicts occur, the commit fails for all transactions that include the `SELECT FOR UPDATE` clause. If no conflict is detected, the commit succeeds. And a new version is generated for the locked rows, so that write conflicts can be detected when other uncommitted transactions are being committed later.| -|`LOCK IN SHARE MODE` | To guarantee compatibility, TiDB parses these three modifiers, but will ignore them.| +| `FOR UPDATE` | The `SELECT FOR UPDATE` clause locks all the data in the result sets to detect concurrent updates from other transactions. Data that match the query conditions but do not exist in the result sets are not read-locked, such as the row data written by other transactions after the current transaction is started. TiDB uses the [Optimistic Transaction Model](/reference/transactions/transaction-optimistic.md). The transaction conflicts are not detected in the statement execution phase. Therefore, the current transaction does not block other transactions from executing `UPDATE`, `DELETE` or `SELECT FOR UPDATE` like other databases such as PostgreSQL. In the committing phase, the rows read by `SELECT FOR UPDATE` are committed in two phases, which means they can also join the conflict detection. If write conflicts occur, the commit fails for all transactions that include the `SELECT FOR UPDATE` clause. If no conflict is detected, the commit succeeds. And a new version is generated for the locked rows, so that write conflicts can be detected when other uncommitted transactions are being committed later. When using pessimistic transaction model, the behavior is basically the same as other databases. Refer to [Difference with MySQL InnoDB](/reference/transactions/transaction-pessimistic.md#difference-with-mysql-innodb) to see the details. | +|`LOCK IN SHARE MODE` | To guarantee compatibility, TiDB parses these three modifiers, but will ignore them. | ## Examples diff --git a/reference/transactions/overview.md b/reference/transactions/overview.md index 4531ee72129b7..1f6a32f55df91 100644 --- a/reference/transactions/overview.md +++ b/reference/transactions/overview.md @@ -1,14 +1,14 @@ --- title: Transactions -summary: Learn how to use the distributed transaction statements. +summary: Learn transactions in TiDB. category: reference --- # Transactions -TiDB supports complete distributed transactions. This document introduces transaction-related statements, explicit and implicit transactions, isolation levels, and laziness checks for transactions. +TiDB supports complete distributed transactions. Both [optimistic transaction model](/reference/transactions/transaction-optimistic.md) and [pessimistic transaction model](/reference/transactions/transaction-pessimistic.md)(introduced in TiDB 3.0) are available. This document introduces transaction-related statements, explicit and implicit transactions, isolation levels, lazy check for constraints, and transaction sizes. -The common variables include `autocommit`, [`tidb_disable_txn_auto_retry`](/reference/configuration/tidb-server/tidb-specific-variables.md#tidb_disable_txn_auto_retry), and [`tidb_retry_limit`](/reference/configuration/tidb-server/tidb-specific-variables.md#tidb_retry_limit). +The common variables include [`autocommit`](#autocommit), [`tidb_disable_txn_auto_retry`](/reference/configuration/tidb-server/tidb-specific-variables.md#tidb_disable_txn_auto_retry), and [`tidb_retry_limit`](/reference/configuration/tidb-server/tidb-specific-variables.md#tidb_retry_limit). ## Common syntax @@ -70,9 +70,9 @@ Syntax: SET autocommit = {0 | 1} ``` -If you set the value of `autocommit` to `1`, the status of the current session is autocommit. If you set the value of `autocommit` to `0`, the status of the current session is non-autocommit. The value of `autocommit` is `1` by default. +When `autocommit = 1` (default), the status of the current session is autocommit. That is, statements are automatically committed immediately following their execution. -When autocommit is enabled, statements are automatically committed immediately following their execution. When autocommit is disabled, statements are only committed when you execute the `COMMIT` statement. +When `autocommit = 0`, the status of the current session is non-autocommit. That is, statements are only committed when you manually execute the `COMMIT` statement. > **Note:** > @@ -94,7 +94,7 @@ SET @@GLOBAL.autocommit = {0 | 1}; ## Explicit and implicit transaction -TiDB supports explicit transactions (`[BEGIN|START TRANSACTION]` and `COMMIT`) and implicit transactions (`SET autocommit = 1`). +TiDB supports explicit transactions (use `[BEGIN|START TRANSACTION]` and `COMMIT` to define the start and end of the transaction) and implicit transactions (`SET autocommit = 1`). If you set the value of `autocommit` to `1` and start a new transaction through the `[BEGIN|START TRANSACTION]` statement, the autocommit is disabled before `COMMIT` or `ROLLBACK` which makes the transaction becomes explicit. @@ -153,3 +153,52 @@ rollback; ``` In the above example, the second `insert` statement fails, and this transaction does not insert any data into the database because `rollback` is called. + +## Transaction sizes + +In TiDB, a transaction either too small or too large can impair the overall performance. + +### Small transactions + +TiDB uses the default autocommit setting (that is, `autocommit = 1`), which automatically issues a commit when executing each SQL statement. Therefore, each of the following three statements is treated as a transaction: + +```sql +UPDATE my_table SET a = 'new_value' WHERE id = 1; +UPDATE my_table SET a = 'newer_value' WHERE id = 2; +UPDATE my_table SET a = 'newest_value' WHERE id = 3; +``` + +In this case, the latency is increased because each statement, as a transaction, uses the two-phase commit which consumes more execution time. + +To improve the execution efficiency, you can use an explicit transaction instead, that is, to execute the above three statements within a transaction: + +```sql +START TRANSACTION; +UPDATE my_table SET a = 'new_value' WHERE id = 1; +UPDATE my_table SET a = 'newer_value' WHERE id = 2; +UPDATE my_table SET a = 'newest_value' WHERE id = 3; +COMMIT; +``` + +Similarly, it is recommended to execute `INSERT` statements within an explicit transaction. + +> **Note:** +> +> The single-threaded workloads in TiDB might not fully use TiDB's distributed resources, so the performance of TiDB is lower than that of a single-instance deployment of MySQL. This difference is similar to the case of transactions with higher latency in TiDB. + +### Large transaction + +Due to the requirement of the two-phase commit, a large transaction can lead to the following issues: + +* OOM (Out of Memory) when excessive data is written in the memory +* More conflicts in the prewrite phase +* Long duration before transactions are actually committed + +Therefore, TiDB intentionally imposes some limits on transaction sizes: + +* The total number of SQL statements in a transaction is no more than 5,000 (default) +* Each key-value pair is no more than 6 MB + +For each transaction, it is recommended to keep the number of SQL statements between 100 to 500 to achieve an optimal performance. + +TiDB sets a default limit of 100 MB for the total size of key-value pairs, which can be modified by the `txn-total-size-limit` configuration item in the configuration file. The maximum value of `txn-total-size-limit` is 10 GB. The actual size limit of one transaction also depends on the memory capacity. When executing large transactions, the memory usage of the TiDB process is approximately 6 times larger than the total size of transactions. diff --git a/reference/transactions/transaction-isolation.md b/reference/transactions/transaction-isolation.md index bdc366c1c84e5..c70d73013800c 100644 --- a/reference/transactions/transaction-isolation.md +++ b/reference/transactions/transaction-isolation.md @@ -6,7 +6,7 @@ category: reference # TiDB Transaction Isolation Levels -Transaction isolation is one of the foundations of database transaction processing. Isolation is the I in the acronym ACID (Atomicity, Consistency, Isolation, Durability), which represents the isolation property of database transactions. +Transaction isolation is one of the foundations of database transaction processing. Isolation is one of the four key properties of a transaction (commonly referred as [ACID](/glossary.md#acid)). The SQL-92 standard defines four levels of transaction isolation: Read Uncommitted, Read Committed, Repeatable Read, and Serializable. See the following table for details: @@ -17,11 +17,11 @@ The SQL-92 standard defines four levels of transaction isolation: Read Uncommitt | REPEATABLE READ | Not Possible | Not possible | Not possible | Possible | | SERIALIZABLE | Not Possible | Not possible | Not possible | Not possible | -TiDB implements Snapshot Isolation consistency, which it advertises as `REPEATABLE-READ` for compatibility with MySQL. This differs from the [ANSI Repeatable Read isolation level](#difference-between-tidb-and-ansi-repeatable-read) and the [MySQL Repeatable Read level](#difference-between-tidb-and-mysql-repeatable-read). +TiDB implements Snapshot Isolation (SI) consistency, which it advertises as `REPEATABLE-READ` for compatibility with MySQL. This differs from the [ANSI Repeatable Read isolation level](#difference-between-tidb-and-ansi-repeatable-read) and the [MySQL Repeatable Read level](#difference-between-tidb-and-mysql-repeatable-read). > **Note:** > -> In the default configuration of TiDB v3.0, the automatic transaction retry is disabled. For how this feature influences the isolation level and how to enable it, see [automatic retry and transactional anomalies caused by automatic retry](#automatic-retry-and-transactional-anomalies-caused-by-automatic-retry). +> In the default configuration of TiDB v3.0, the automatic transaction retry is disabled. For how this feature influences the isolation level and how to enable it, see [automatic retry](/reference/transactions/transaction-optimistic.md#automatic-retry). TiDB uses the [Percolator transaction model](https://research.google.com/pubs/pub36726.html). A global read timestamp is obtained when the transaction is started, and a global commit timestamp is obtained when the transaction is committed. The execution order of transactions is confirmed based on the timestamps. To know more about the implementation of TiDB transaction model, see [MVCC in TiKV](https://pingcap.com/blog/2016-11-17-mvcc-in-tikv/). @@ -46,7 +46,7 @@ commit; | ### Difference between TiDB and ANSI Repeatable Read -The Repeatable Read isolation level in TiDB differs from ANSI Repeatable Read isolation level, though they sharing the same name. According to the standard described in the [A Critique of ANSI SQL Isolation Levels](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-95-51.pdf) paper, TiDB implements the Snapshot Isolation (SI) level. This isolation level does not allow strict phantoms (A3) but allows broad phantoms (P3) and write skews. In contrast, the ANSI Repeatable Read isolation level allows phantom reads but does not allow write skews. +The Repeatable Read isolation level in TiDB differs from ANSI Repeatable Read isolation level, though they sharing the same name. According to the standard described in the [A Critique of ANSI SQL Isolation Levels](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-95-51.pdf) paper, TiDB implements the Snapshot Isolation level. This isolation level does not allow strict phantoms (A3) but allows broad phantoms (P3) and write skews. In contrast, the ANSI Repeatable Read isolation level allows phantom reads but does not allow write skews. ### Difference between TiDB and MySQL Repeatable Read @@ -63,53 +63,3 @@ For historical reasons, the Read Committed isolation level of current mainstream ## Difference between TiDB and MySQL Read Committed The MySQL Read Committed isolation level is in line with the Consistent Read features in most cases. There are also exceptions, such as [semi-consistent read](https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html). This special behavior is not supported in TiDB. - -## Automatic retry and transactional anomalies caused by automatic retry - -By default TiDB will not retry transactions because this might lead to lost updates and damaged Snapshot Isolation. If your application can tolerate lost updates, and does not require Snapshot Isolation consistency, you can enable **this feature** by setting `tidb_disable_txn_auto_retry = off`. This has the benefit of fewer `COMMIT` statements generating errors. The value of `tidb_retry_limit` cannot be `0`. Otherwise, the automatic retry is also disabled. - -When automatic retry is enabled, conflicts in an explicit transaction might lead to unexpected result. - -Example 1: - -| Session1 | Session2 | -| ---------------- | ------------ | -| `begin;` | `begin;` | -| `select balance from t where id = 1;` | `update t set balance = balance -100 where id = 1;` | -| | `update t set balance = balance -100 where id = 2;` | -| // the subsequent logic depends on the result of `select` | `commit;` | -| `if balance > 100 {` | | -| `update t set balance = balance + 100 where id = 2;` | | -| `}` | | -| `commit;` // automatic retry | | - -Example 2: - -| Session1 | Session2 | -| ---------------- | ------------ | -| `begin;` | `begin;` | -| `update t set balance = balance - 100 where id = 1;` | `delete from t where id = 1;` | -| | `commit;` | -| // the subsequent logic depends on the result of `affected_rows` | | -| `if affected_rows > 0 {` | | -| `update t set balance = balance + 100 where id = 2;` | | -| `}` | | -| `commit;` // automatic retry | | - -Under the automatic retry mechanism of TiDB, all the executed statements for the first time are re-executed again. Whether the subsequent statements are to be executed or not depends on the results of the previous statements, automatic retry can violate snapshot isolation, causing lost updates. - -To disable the automatic retry of explicit transactions, configure `tidb_disable_txn_auto_retry = on`: - -{{< copyable "sql" >}} - -```sql -SET GLOBAL tidb_disable_txn_auto_retry = on; -``` - -Changing the variable `tidb_disable_txn_auto_retry` does not affect the implicit single statement with `autocommit = 1`, because the automatic retry of this statement does not cause anomalies like update loss and does not break the isolation of a transaction. - -After the automatic retry of explicit transactions is disabled, if a transaction conflict occurs, the `commit` statement returns an error that includes the `try again later` string. The application layer uses this string to judge whether the error can be retried. - -If the application layer logic is included in the process of transaction execution, it is recommended to add the retry of explicit transactions at the application layer and disable automatic retry. - -The `tidb_retry_limit` variable determines the maximum number of transaction retries. The default value is 10. When this variable is set to 0, none of the transactions automatically retries, including the implicit single statement transactions that are automatically committed. This is the way to completely disable the automatic retry mechanism in TiDB. If you are more concerned with the latency of transaction execution than transaction isolation, you can set it to 0. Then all conflicting transactions report failures to the application layer in the fastest way. diff --git a/reference/transactions/transaction-model.md b/reference/transactions/transaction-model.md deleted file mode 100644 index a38ec36169cdf..0000000000000 --- a/reference/transactions/transaction-model.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Transaction Model -summary: Learn TiDB's transaction model and its differences with MySQL. -category: reference ---- - -# Transaction Model - -TiDB implements an optimistic transaction model. Unlike MySQL, which uses row-level locking to avoid write conflict, in TiDB, the write conflict is checked only in the `commit` process during the execution of the statements like `Update`, `Insert`, `Delete`, and so on. - -Similarly, statements such as `SELECT .. FOR UPDATE` do not work in the same way as in MySQL. Therefore, it is important to check the returned results of `commit` statements, because even if there is no error in execution, there might be errors during the `COMMIT` process. - -> **Note:** -> -> Experimental support for [pessimistic locking](/reference/transactions/transaction-pessimistic.md) is now available. When enabled, TiDB will behave behave similar to the InnoDB storage engine. - -## Transaction restrictions - -Due to the distributed, 2-phase commit requirement of TiDB, large transactions that modify data can be particularly problematic. TiDB intentionally sets some limits on transaction sizes to reduce this impact: - -* A transaction is limited to 5000 SQL statements (by default) -* Each Key-Value entry is no more than 6MB -* The total size of Key-Value entries is no more than 100MB - -### Best practices - -Because each transaction in TiDB requires two round trips to the PD leader, small transactions might have higher latencies in TiDB than MySQL. As a hypothetical example, the following query could be improved by moving from `autocommit` to using an explicit transaction: - -The original version with `autocommit`: - -{{< copyable "sql" >}} - -```sql -UPDATE my_table SET a='new_value' WHERE id = 1; -UPDATE my_table SET a='newer_value' WHERE id = 2; -UPDATE my_table SET a='newest_value' WHERE id = 3; -``` - -The improved version: - -{{< copyable "sql" >}} - -```sql -START TRANSACTION; -UPDATE my_table SET a='new_value' WHERE id = 1; -UPDATE my_table SET a='newer_value' WHERE id = 2; -UPDATE my_table SET a='newest_value' WHERE id = 3; -COMMIT; -``` - -> **Note:** -> -> The distributed resources in TiDB might not be fully used in the single-threaded workloads, so the performance of TiDB is lower than that of a single-instance deployment of MySQL. This difference is similar to the case of transactions with higher latency in TiDB. - -### SELECT .. FOR UPDATE - -Due to optimistic locking, `SELECT .. FOR UPDATE` statements do not block other sessions from modifying data. Instead, the `SELECT .. FOR UPDATE` statement will cause the transaction to fail if rows have been modified by another transaction. Similarly, the `SELECT .. FOR UPDATE` statement disables any transaction retry. diff --git a/reference/transactions/transaction-optimistic.md b/reference/transactions/transaction-optimistic.md new file mode 100644 index 0000000000000..aa2188bd9467b --- /dev/null +++ b/reference/transactions/transaction-optimistic.md @@ -0,0 +1,175 @@ +--- +title: TiDB Optimistic Transaction Model +summary: Learn the optimistic transaction model in TiDB. +category: reference +aliases: ['/docs/dev/reference/transactions/transaction-model/'] +--- + +# TiDB Optimistic Transaction Model + +This document introduces the principles of TiDB's optimistic transaction model. This document assumes that you have a basic understanding of [TiDB architecture](/architecture.md), [Percolator](https://ai.google/research/pubs/pub36726), and the [ACID](/glossary.md#acid) properties of transactions. + +In TiDB's optimistic transaction model, the two-phase commit begins right after the client executes the `COMMIT` statement. Therefore, the write-write conflict can be observed before the transactions are actually committed. + +> **Note:** +> +> Starting from v3.0.8, TiDB uses the [pessimistic transaction model](/reference/transactions/transaction-pessimistic.md) by default. However, this does not affect your clusters if you upgrading from v3.0.7 or earlier to v3.0.8 (and later). In other words, **only newly created clusters default to using the pessimistic transaction model**. + +## Principles of optimistic transactions + +TiDB adopts Google's Percolator transaction model, a variant of two-phase commit (2PC) to ensure the correct completion of a distributed transaction. The procedure is as follows: + +![2PC in TiDB](/media/2pc-in-tidb.png) + +1. The client begins a transaction. + + TiDB receives the start version number (monotonically increasing in time and globally unique) from PD and mark it as `start_ts`. + +2. The client issues a read request. + 1. TiDB receives routing information (how data is distributed among TiKV nodes) from PD. + 2. TiDB receives the data of the `start_ts` version from TiKV. + +3. The client issues a write request. + + TiDB checks whether the written data satisfies consistency constraints (to ensure the data types are correct and the unique index is met etc.) **Valid data is stored in the memory**. + +4. The client issues a commit request. + +5. TiDB begins 2PC to ensure the atomicity of distributed transactions and persist data in store. + + 1. TiDB selects a Primary Key from the data to be written. + 2. TiDB receives the information of Region distribution from PD, and groups all keys by Region accordingly. + 3. TiDB sends prewrite requests to all TiKV nodes involved. Then, TiKV checks whether there are conflict or expired versions. Valid data is locked. + 4. TiDB receives all requests in the prewrite phase and the prewrite is successful. + 5. TiDB receives a commit version number from PD and marks it as `commit_ts`. + 6. TiDB initiates the second commit to the TiKV node where Primary Key is located. TiKV checks the data, and clean the locks left in the prewrite phase. + 7. TiDB receives the message that reports the second phase is successfully finished. + +6. TiDB returns a message to inform the client that the transaction is successfully committed. + +7. TiDB asynchronously cleans the locks left in this transaction. + +## Advantages and disadvantages + +From the process of transactions in TiDB above, it is clear that TiDB transactions have the following advantages: + +* Simple to understand +* Implement cross-node transaction based on single-row transaction +* Decentralized lock management + +However, TiDB transactions also have the following disadvantages: + +* Transaction latency due to 2PC +* In need of a centralized version manager +* OOM (out of memory) when extensive data is written in the memory + +To avoid potential problems in application, refer to [transaction sizes](/reference/transactions/overview.md#transaction-size) to see more details. + +## Transaction retries + +TiDB uses optimistic concurrency control by default whereas MySQL applies pessimistic concurrency control. This means that MySQL checks for conflicts during the execution of SQL statements, so there are few errors reported in heavy contention scenarios. For the convenience of MySQL users, TiDB provides a retry function that runs inside a transaction. + +### Automatic retry + +If there is a conflict, TiDB retries the write operations automatically. You can set `tidb_disable_txn_auto_retry` and `tidb_retry_limit` to enable or disable this default function: + +```toml +# Whether to disable automatic retry. ("on" by default) +tidb_disable_txn_auto_retry = off +# Set the maximum number of the retires. ("10" by default) +# When “tidb_retry_limit = 0”, automatic retry is completely disabled. +tidb_retry_limit = 10 +``` + +You can enable the automatic retry in either session level or global level: + +1. Session level: + + {{< copyable "sql" >}} + + ```sql + set @@tidb_disable_txn_auto_retry = off; + ``` + + {{< copyable "sql" >}} + + ```sql + set @@tidb_retry_limit = 10; + ``` + +2. Global level: + + {{< copyable "sql" >}} + + ```sql + set @@global.tidb_disable_txn_auto_retry = off; + ``` + + {{< copyable "sql" >}} + + ```sql + set @@global.tidb_retry_limit = 10; + ``` + +> **Note:** +> +> The `tidb_retry_limit` variable decides the maximum number of retries. When this variable is set to `0`, none of the transactions automatically retries, including the implicit single statement transactions that are automatically committed. This is the way to completely disable the automatic retry mechanism in TiDB. After the automatic retry is disabled, all conflicting transactions report failures (includes the `try again later` string) to the application layer in the fastest way. + +### Limits of retry + +By default, TiDB will not retry transactions because this might lead to lost updates and damaged [`REPEATABLE READ` isolation](/reference/transactions/transaction-isolation.md). + +The reason can be observed from the procedures of retry: + +1. Allocate a new timestamp and mark it as `start_ts`. +2. Retry the SQL statements that contain write operations. +3. Implement the two-phase commit. + +In Step 2, TiDB only retries SQL statements that contain write operations. However, during retrying, TiDB receives a new version number to mark the beginning of the transaction. This means that TiDB retries SQL statements with the data in the new `start_ts` version. In this case, if the transaction updates data using other query results, the results might be inconsistent because the `REPEATABLE READ` isolation is violated. + +If your application can tolerate lost updates, and does not require `REPEATABLE READ` isolation consistency, you can enable this feature by setting `tidb_disable_txn_auto_retry = off`. + +## Conflict detection + +For the optimistic transaction, it is important to detect whether there are write-write conflicts in the underlying data. Although TiKV reads data for detection **in the prewrite phase**, a conflict pre-detection is also performed in the TiDB clusters to improve the efficiency. + +Because TiDB is a distributed database, the conflict detection in the memory is performed in two layers: + +* The TiDB layer. If a write-write conflict in the instance is observed after the primary write is issued, it is unnecessary to issue the subsequent writes to the TiKV layer. +* The TiKV layer. TiDB instances are unaware of each other, which means they cannot confirm whether there are conflicts or not. Therefore, the conflict detection is mainly performed in the TiKV layer. + +The conflict detection in the TiDB layer is disabled by default. The specific configuration items are as follows: + +```toml +[txn-local-latches] +# Whether to enable the latches for transactions. Recommended +# to use latches when there are many local transaction conflicts. +# ("false" by default) +enabled = false +# Controls the number of slots corresponding to Hash. ("204800" by default) +# It automatically adjusts upward to an exponential multiple of 2. +# Each slot occupies 32 Bytes of memory. If set too small, +# it might result in slower running speed and poor performance +# when data writing covers a relatively large range. +capacity = 2048000 +``` + +The value of the `capacity` configuration item mainly affects the accuracy of conflict detection. During conflict detection, only the hash value of each key is stored in the memory. Because the probability of collision when hashing is closely related to the probability of misdetection, you can configure `capacity` to controls the number of slots and enhance the accuracy of conflict detection. + +* The smaller the value of `capacity`, the smaller the occupied memory and the greater the probability of misdetection. +* The larger the value of `capacity`, the larger the occupied memory and the smaller the probability of misdetection. + +When you confirm that there is no write-write conflict in the upcoming transactions (such as importing data), it is recommended to disable the function of conflict detection. + +TiKV also uses a similar mechanism to detect conflicts, but the conflict detection in the TiKV layer cannot be disabled. You can only configure `scheduler-concurrency` to control the number of slots that defined by the modulo operation: + +```toml +# Controls the number of slots. ("2048000" by default） +scheduler-concurrency = 2048000 +``` + +In addition, TiKV supports monitoring the time spent on waiting latches in scheduler. + +![Scheduler latch wait duration](/media/optimistic-transaction-metric.png) + +When `Scheduler latch wait duration` is high and there is no slow writes, it can be safely concluded that there are many write conflicts at this time. diff --git a/reference/transactions/transaction-pessimistic.md b/reference/transactions/transaction-pessimistic.md index 34a3c6f104133..a0ab1ffdc4486 100644 --- a/reference/transactions/transaction-pessimistic.md +++ b/reference/transactions/transaction-pessimistic.md @@ -1,12 +1,12 @@ --- -title: TiDB Pessimistic Transaction Mode -summary: Learn about the pessimistic transaction mode of TiDB. +title: TiDB Pessimistic Transaction Model +summary: Learn the pessimistic transaction model in TiDB. category: reference --- -# TiDB Pessimistic Transaction Mode +# TiDB Pessimistic Transaction Model -By default, TiDB implements the optimistic transaction mode, where the transaction commit might fail because of transaction conflicts. To make sure that the commit succeeds, you need to modify the application and add an automatic retry mechanism. You can avoid this issue by using the pessimistic transaction mode of TiDB. +An optimistic transaction in TiDB might fail to commit because of transaction conflicts. To make sure that the commit succeeds, you need to modify the application and add an automatic retry mechanism. Using the pessimistic transaction model can avoid this issue. ## Behaviors of the pessimistic transaction mode @@ -58,4 +58,4 @@ To disable the pessimistic transaction mode, modify the configuration file and a 2. TiDB does not support `SELECT LOCK IN SHARE MODE`. - When `SELECT LOCK IN SHARE MODE` is specified in a statement, it has the same effect as that without the lock, so the read or write of other transactions is not blocked. \ No newline at end of file + When `SELECT LOCK IN SHARE MODE` is specified in a statement, it has the same effect as that without the lock, so the read or write of other transactions is not blocked.