From 8f03b7dae8121b99be63894d5624cc2d07e0da67 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 4 Jun 2024 17:33:29 +0800 Subject: [PATCH 1/7] Add temp.md --- temp.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 temp.md diff --git a/temp.md b/temp.md new file mode 100644 index 0000000000000..af27ff4986a7b --- /dev/null +++ b/temp.md @@ -0,0 +1 @@ +This is a test file. \ No newline at end of file From 7e94d298565ccdd3da3825365e21976d7d4b492b Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 4 Jun 2024 17:33:33 +0800 Subject: [PATCH 2/7] Delete temp.md --- temp.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 temp.md diff --git a/temp.md b/temp.md deleted file mode 100644 index af27ff4986a7b..0000000000000 --- a/temp.md +++ /dev/null @@ -1 +0,0 @@ -This is a test file. \ No newline at end of file From f8ba96cd85047679523b993b2aa5f7beceabfcfc Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Tue, 4 Jun 2024 17:45:15 +0800 Subject: [PATCH 3/7] add translation --- releases/release-8.1.0.md | 2 +- ticdc/ticdc-behavior-change.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-8.1.0.md b/releases/release-8.1.0.md index 76a954360180e..ce23e53747dd4 100644 --- a/releases/release-8.1.0.md +++ b/releases/release-8.1.0.md @@ -172,7 +172,7 @@ Compared with the previous LTS 7.5.0, 8.1.0 includes new features, improvements, * In earlier versions, the `tidb.tls` configuration item in TiDB Lightning treats values `"false"` and `""` the same, as well as treating the values `"preferred"` and `"skip-verify"` the same. Starting from v8.1.0, TiDB Lightning distinguishes the behavior of `"false"`, `""`, `"skip-verify"`, and `"preferred"` for `tidb.tls`. For more information, see [TiDB Lightning configuration](/tidb-lightning/tidb-lightning-configuration.md). * For tables with `AUTO_ID_CACHE=1`, TiDB supports a [centralized auto-increment ID allocating service](/auto-increment.md#mysql-compatibility-mode). In earlier versions, the primary TiDB node of this service automatically performs a `forceRebase` operation when the TiDB process exits (for example, during the TiDB node restart) to keep auto-assigned IDs as consecutive as possible. However, when there are too many tables with `AUTO_ID_CACHE=1`, executing `forceRebase` becomes very time-consuming, preventing TiDB from restarting promptly and even blocking data writes, thus affecting system availability. To resolve this issue, starting from v8.1.0, TiDB removes the `forceRebase` behavior, but this change will cause some auto-assigned IDs to be non-consecutive during the failover. -* In earlier versions, when processing a transaction containing `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event, TiCDC splits this event into `DELETE` and `INSERT` events. Starting from v8.1.0, when using the MySQL sink, TiCDC only splits an `UPDATE` event into `DELETE` and `INSERT` events if the primary key or non-null unique index value is modified in the `UPDATE` event and the transaction `commitTS` is less than TiCDC `thresholdTs` (which is the current timestamp that TiCDC fetches from PD at TiCDC startup). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. For more information, see [documentation](/ticdc/ticdc-behavior-change.md#mysql-sink). +* In earlier versions, when processing a transaction containing `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event, TiCDC splits this event into `DELETE` and `INSERT` events. Starting from v8.1.0, when using the MySQL sink, TiCDC splits an `UPDATE` event into `DELETE` and `INSERT` events if the transaction `commitTS` for the `UPDATE` change is less than TiCDC `thresholdTs` (which is the current timestamp that TiCDC fetches from PD at TiCDC startup). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. For more information, see [documentation](/ticdc/ticdc-behavior-change.md#mysql-sink). ### System variables diff --git a/ticdc/ticdc-behavior-change.md b/ticdc/ticdc-behavior-change.md index 317b971071531..4c82024158ed3 100644 --- a/ticdc/ticdc-behavior-change.md +++ b/ticdc/ticdc-behavior-change.md @@ -57,7 +57,7 @@ Therefore, TiCDC splits these two events into four events, that is, deleting rec Starting from v8.1.0, when using the MySQL sink, TiCDC fetches the current timestamp `thresholdTs` from PD at startup and decides whether to split `UPDATE` events based on the value of this timestamp: -- For transactions containing one or multiple `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event and the transaction `commitTS` is less than `thresholdTs`, TiCDC splits the `UPDATE` event into a `DELETE` event and an `INSERT` event before writing them to the Sorter module. +- For transactions containing one or multiple `UPDATE` changes, if the transaction `commitTS` is less than `thresholdTs`, TiCDC splits the `UPDATE` event into a `DELETE` event and an `INSERT` event before writing them to the Sorter module. - For `UPDATE` events with the transaction `commitTS` greater than or equal to `thresholdTs`, TiCDC does not split them. For more information, see GitHub issue [#10918](https://github.com/pingcap/tiflow/issues/10918). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. From d7899c7ac604c1b48d4cd9d1cc1dd0b94022ab9d Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 5 Jun 2024 15:59:19 +0800 Subject: [PATCH 4/7] Update release-8.1.0.md --- releases/release-8.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.1.0.md b/releases/release-8.1.0.md index ce23e53747dd4..5947f627eb595 100644 --- a/releases/release-8.1.0.md +++ b/releases/release-8.1.0.md @@ -172,7 +172,7 @@ Compared with the previous LTS 7.5.0, 8.1.0 includes new features, improvements, * In earlier versions, the `tidb.tls` configuration item in TiDB Lightning treats values `"false"` and `""` the same, as well as treating the values `"preferred"` and `"skip-verify"` the same. Starting from v8.1.0, TiDB Lightning distinguishes the behavior of `"false"`, `""`, `"skip-verify"`, and `"preferred"` for `tidb.tls`. For more information, see [TiDB Lightning configuration](/tidb-lightning/tidb-lightning-configuration.md). * For tables with `AUTO_ID_CACHE=1`, TiDB supports a [centralized auto-increment ID allocating service](/auto-increment.md#mysql-compatibility-mode). In earlier versions, the primary TiDB node of this service automatically performs a `forceRebase` operation when the TiDB process exits (for example, during the TiDB node restart) to keep auto-assigned IDs as consecutive as possible. However, when there are too many tables with `AUTO_ID_CACHE=1`, executing `forceRebase` becomes very time-consuming, preventing TiDB from restarting promptly and even blocking data writes, thus affecting system availability. To resolve this issue, starting from v8.1.0, TiDB removes the `forceRebase` behavior, but this change will cause some auto-assigned IDs to be non-consecutive during the failover. -* In earlier versions, when processing a transaction containing `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event, TiCDC splits this event into `DELETE` and `INSERT` events. Starting from v8.1.0, when using the MySQL sink, TiCDC splits an `UPDATE` event into `DELETE` and `INSERT` events if the transaction `commitTS` for the `UPDATE` change is less than TiCDC `thresholdTs` (which is the current timestamp that TiCDC fetches from PD at TiCDC startup). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. For more information, see [documentation](/ticdc/ticdc-behavior-change.md#mysql-sink). +* In earlier versions, when processing a transaction containing `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event, TiCDC splits this event into `DELETE` and `INSERT` events. Starting from v8.1.0, when using the MySQL sink, TiCDC splits an `UPDATE` event into `DELETE` and `INSERT` events if the transaction `commitTS` for the `UPDATE` change is less than TiCDC `thresholdTS` (which is the current timestamp that TiCDC fetches from PD at TiCDC startup). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. For more information, see [documentation](/ticdc/ticdc-behavior-change.md#mysql-sink). ### System variables From df70b3420c54a752adc9735a3cd6180c56729441 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 5 Jun 2024 15:59:55 +0800 Subject: [PATCH 5/7] thresholdTs -> thresholdTS --- ticdc/ticdc-behavior-change.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/ticdc/ticdc-behavior-change.md b/ticdc/ticdc-behavior-change.md index 4c82024158ed3..59bb61d4b23a7 100644 --- a/ticdc/ticdc-behavior-change.md +++ b/ticdc/ticdc-behavior-change.md @@ -55,10 +55,10 @@ Therefore, TiCDC splits these two events into four events, that is, deleting rec ### MySQL sink -Starting from v8.1.0, when using the MySQL sink, TiCDC fetches the current timestamp `thresholdTs` from PD at startup and decides whether to split `UPDATE` events based on the value of this timestamp: +Starting from v8.1.0, when using the MySQL sink, TiCDC fetches the current timestamp `thresholdTS` from PD at startup and decides whether to split `UPDATE` events based on the value of this timestamp: -- For transactions containing one or multiple `UPDATE` changes, if the transaction `commitTS` is less than `thresholdTs`, TiCDC splits the `UPDATE` event into a `DELETE` event and an `INSERT` event before writing them to the Sorter module. -- For `UPDATE` events with the transaction `commitTS` greater than or equal to `thresholdTs`, TiCDC does not split them. For more information, see GitHub issue [#10918](https://github.com/pingcap/tiflow/issues/10918). +- For transactions containing one or multiple `UPDATE` changes, if the transaction `commitTS` is less than `thresholdTS`, TiCDC splits the `UPDATE` event into a `DELETE` event and an `INSERT` event before writing them to the Sorter module. +- For `UPDATE` events with the transaction `commitTS` greater than or equal to `thresholdTS`, TiCDC does not split them. For more information, see GitHub issue [#10918](https://github.com/pingcap/tiflow/issues/10918). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. @@ -97,7 +97,7 @@ UPDATE t SET a = 3 WHERE a = 2; After the downstream executes the transaction, the records in the database are `(3, 2)`, which are different from the records in the upstream database (`(2, 1)` and `(3, 2)`), indicating a data inconsistency issue. -- After this behavior change, if the transaction `commitTS` is less than the `thresholdTs` obtained by TiCDC at startup, TiCDC splits these `UPDATE` events into `DELETE` and `INSERT` events before writing them to the Sorter module. After the sorting by the Sorter module, the actual execution order of these events in the downstream is as follows: +- After this behavior change, if the transaction `commitTS` is less than the `thresholdTS` obtained by TiCDC at startup, TiCDC splits these `UPDATE` events into `DELETE` and `INSERT` events before writing them to the Sorter module. After the sorting by the Sorter module, the actual execution order of these events in the downstream is as follows: ```sql BEGIN; @@ -114,4 +114,4 @@ As you can see from the preceding example, splitting the `UPDATE` event into `DE > **Note:** > -> After this behavior change, when using the MySQL sink, TiCDC does not split the `UPDATE` event in most cases. Consequently, there might be primary key or unique key conflicts during changefeed runtime, causing the changefeed to restart automatically. After the restart, TiCDC will split the conflicting `UPDATE` events into `DELETE` and `INSERT` events before writing them to the Sorter module. This ensures that all events within the same transaction are correctly ordered, with all `DELETE` events preceding `INSERT` events, thus correctly completing data replication. \ No newline at end of file +> After this behavior change, when using the MySQL sink, TiCDC does not split the `UPDATE` event in most cases. Consequently, there might be primary key or unique key conflicts during changefeed runtime, causing the changefeed to restart automatically. After the restart, TiCDC will split the conflicting `UPDATE` events into `DELETE` and `INSERT` events before writing them to the Sorter module. This ensures that all events within the same transaction are correctly ordered, with all `DELETE` events preceding `INSERT` events, thus correctly completing data replication. From d1278790b133684750c871b584b664c5c2ef1d44 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 17 Jun 2024 15:44:33 +0800 Subject: [PATCH 6/7] Update releases/release-8.1.0.md --- releases/release-8.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.1.0.md b/releases/release-8.1.0.md index 5947f627eb595..e0233618ed842 100644 --- a/releases/release-8.1.0.md +++ b/releases/release-8.1.0.md @@ -172,7 +172,7 @@ Compared with the previous LTS 7.5.0, 8.1.0 includes new features, improvements, * In earlier versions, the `tidb.tls` configuration item in TiDB Lightning treats values `"false"` and `""` the same, as well as treating the values `"preferred"` and `"skip-verify"` the same. Starting from v8.1.0, TiDB Lightning distinguishes the behavior of `"false"`, `""`, `"skip-verify"`, and `"preferred"` for `tidb.tls`. For more information, see [TiDB Lightning configuration](/tidb-lightning/tidb-lightning-configuration.md). * For tables with `AUTO_ID_CACHE=1`, TiDB supports a [centralized auto-increment ID allocating service](/auto-increment.md#mysql-compatibility-mode). In earlier versions, the primary TiDB node of this service automatically performs a `forceRebase` operation when the TiDB process exits (for example, during the TiDB node restart) to keep auto-assigned IDs as consecutive as possible. However, when there are too many tables with `AUTO_ID_CACHE=1`, executing `forceRebase` becomes very time-consuming, preventing TiDB from restarting promptly and even blocking data writes, thus affecting system availability. To resolve this issue, starting from v8.1.0, TiDB removes the `forceRebase` behavior, but this change will cause some auto-assigned IDs to be non-consecutive during the failover. -* In earlier versions, when processing a transaction containing `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event, TiCDC splits this event into `DELETE` and `INSERT` events. Starting from v8.1.0, when using the MySQL sink, TiCDC splits an `UPDATE` event into `DELETE` and `INSERT` events if the transaction `commitTS` for the `UPDATE` change is less than TiCDC `thresholdTS` (which is the current timestamp that TiCDC fetches from PD at TiCDC startup). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. For more information, see [documentation](/ticdc/ticdc-behavior-change.md#mysql-sink). +* In earlier versions, when processing a transaction containing `UPDATE` changes, if the primary key or non-null unique index value is modified in an `UPDATE` event, TiCDC splits this event into `DELETE` and `INSERT` events. In v8.1.0, when using the MySQL sink, TiCDC splits an `UPDATE` event into `DELETE` and `INSERT` events if the transaction `commitTS` for the `UPDATE` change is less than TiCDC `thresholdTS` (which is the current timestamp that TiCDC fetches from PD at TiCDC startup). This behavior change addresses the issue of downstream data inconsistencies caused by the potentially incorrect order of `UPDATE` events received by TiCDC, which can lead to an incorrect order of split `DELETE` and `INSERT` events. For more information, see [documentation](/ticdc/ticdc-behavior-change.md#mysql-sink). ### System variables From 541b65b310b9c4e8908eafea818fd4ebfc163e85 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 17 Jun 2024 15:45:24 +0800 Subject: [PATCH 7/7] Update ticdc/ticdc-behavior-change.md --- ticdc/ticdc-behavior-change.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-behavior-change.md b/ticdc/ticdc-behavior-change.md index 59bb61d4b23a7..1689611febe61 100644 --- a/ticdc/ticdc-behavior-change.md +++ b/ticdc/ticdc-behavior-change.md @@ -55,7 +55,7 @@ Therefore, TiCDC splits these two events into four events, that is, deleting rec ### MySQL sink -Starting from v8.1.0, when using the MySQL sink, TiCDC fetches the current timestamp `thresholdTS` from PD at startup and decides whether to split `UPDATE` events based on the value of this timestamp: +In v8.1.0, when using the MySQL sink, TiCDC fetches the current timestamp `thresholdTS` from PD at startup and decides whether to split `UPDATE` events based on the value of this timestamp: - For transactions containing one or multiple `UPDATE` changes, if the transaction `commitTS` is less than `thresholdTS`, TiCDC splits the `UPDATE` event into a `DELETE` event and an `INSERT` event before writing them to the Sorter module. - For `UPDATE` events with the transaction `commitTS` greater than or equal to `thresholdTS`, TiCDC does not split them. For more information, see GitHub issue [#10918](https://github.com/pingcap/tiflow/issues/10918).