From 8f2004921261704607b078a44e4cab9aefe2d11c Mon Sep 17 00:00:00 2001 From: Jianyuan Jiang Date: Thu, 7 Dec 2023 15:11:30 +0800 Subject: [PATCH 1/9] Add temp.md --- temp.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 temp.md diff --git a/temp.md b/temp.md new file mode 100644 index 0000000000000..af27ff4986a7b --- /dev/null +++ b/temp.md @@ -0,0 +1 @@ +This is a test file. \ No newline at end of file From 46ce5169bbd70d7c2a55f4b67dc50f1dd0124d98 Mon Sep 17 00:00:00 2001 From: Jianyuan Jiang Date: Thu, 7 Dec 2023 15:11:36 +0800 Subject: [PATCH 2/9] Delete temp.md --- temp.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 temp.md diff --git a/temp.md b/temp.md deleted file mode 100644 index af27ff4986a7b..0000000000000 --- a/temp.md +++ /dev/null @@ -1 +0,0 @@ -This is a test file. \ No newline at end of file From f8724fd2d76bae12221ee2a3b740dae33cdbb6f3 Mon Sep 17 00:00:00 2001 From: jiangjianyuan Date: Thu, 7 Dec 2023 15:16:49 +0800 Subject: [PATCH 3/9] add cdc redo configuration --- ticdc/ticdc-changefeed-config.md | 3 +++ ticdc/ticdc-open-api-v2.md | 20 +++++++++++--------- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/ticdc/ticdc-changefeed-config.md b/ticdc/ticdc-changefeed-config.md index 6274f775d9c63..920905e29acc3 100644 --- a/ticdc/ticdc-changefeed-config.md +++ b/ticdc/ticdc-changefeed-config.md @@ -192,6 +192,9 @@ flush-worker-num = 8 # The behavior to compress redo log files. # Available options are "" and "lz4". The default value is "", which means no compression. compression = "" +# The concurrency for uploading a single redo file. +# The default value is 1, which means concurrency is disabled. +flush-concurrency = 1 # The following configuration items only take effect when the downstream is Kafka. Supported starting from v6.5.3. [sink.kafka-config] diff --git a/ticdc/ticdc-open-api-v2.md b/ticdc/ticdc-open-api-v2.md index ff0015b67a498..231bbc52b1fbe 100644 --- a/ticdc/ticdc-open-api-v2.md +++ b/ticdc/ticdc-open-api-v2.md @@ -283,15 +283,17 @@ The descriptions of the `replica_config` parameters are as follows. The `consistent` parameters are described as follows: -| Parameter name | Description | -|:-----------------|:---------------------------------------| -| `flush_interval` | `UINT64` type. The interval to flush redo log files. (Optional) | -| `level` | `STRING` type. The consistency level of the replicated data. (Optional) | -| `max_log_size` | `UINT64` type. The maximum value of redo log. (Optional) | -| `storage` | `STRING` type. The destination address of the storage. (Optional) | -| `use_file_backend` | `BOOL` type. Specifies whether to store the redo log in a local file. (Optional) | -| `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | -| `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | +| Parameter name | Description | +|:----------------------|:--------------------------------------------------------------------------------------------------------| +| `flush_interval` | `UINT64` type. The interval to flush redo log files. (Optional) | +| `level` | `STRING` type. The consistency level of the replicated data. (Optional) | +| `max_log_size` | `UINT64` type. The maximum value of redo log. (Optional) | +| `storage` | `STRING` type. The destination address of the storage. (Optional) | +| `use_file_backend` | `BOOL` type. Specifies whether to store the redo log in a local file. (Optional) | +| `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | +| `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | +| `compression` | `STRING` type. The behavior to compress redo log files, available options are "" and "lz4".. (Optional) | +| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. (Optional) | The `filter` parameters are described as follows: From 7c467e80655da1f0118e9d3ebfc95145fcebfb0a Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 7 Dec 2023 17:39:14 +0800 Subject: [PATCH 4/9] rollback the table format changes --- ticdc/ticdc-open-api-v2.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/ticdc/ticdc-open-api-v2.md b/ticdc/ticdc-open-api-v2.md index 231bbc52b1fbe..c2fad7742f4f8 100644 --- a/ticdc/ticdc-open-api-v2.md +++ b/ticdc/ticdc-open-api-v2.md @@ -283,15 +283,15 @@ The descriptions of the `replica_config` parameters are as follows. The `consistent` parameters are described as follows: -| Parameter name | Description | -|:----------------------|:--------------------------------------------------------------------------------------------------------| -| `flush_interval` | `UINT64` type. The interval to flush redo log files. (Optional) | -| `level` | `STRING` type. The consistency level of the replicated data. (Optional) | -| `max_log_size` | `UINT64` type. The maximum value of redo log. (Optional) | -| `storage` | `STRING` type. The destination address of the storage. (Optional) | -| `use_file_backend` | `BOOL` type. Specifies whether to store the redo log in a local file. (Optional) | -| `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | -| `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | +| Parameter name | Description | +|:-----------------|:---------------------------------------| +| `flush_interval` | `UINT64` type. The interval to flush redo log files. (Optional) | +| `level` | `STRING` type. The consistency level of the replicated data. (Optional) | +| `max_log_size` | `UINT64` type. The maximum value of redo log. (Optional) | +| `storage` | `STRING` type. The destination address of the storage. (Optional) | +| `use_file_backend` | `BOOL` type. Specifies whether to store the redo log in a local file. (Optional) | +| `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | +| `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | | `compression` | `STRING` type. The behavior to compress redo log files, available options are "" and "lz4".. (Optional) | | `flush_concurrency` | `INT` type. The concurrency for uploading a single file. (Optional) | From 8b2ae0d2e5c3b5d8da9287f17f3e1dd5462a60e6 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 7 Dec 2023 17:40:05 +0800 Subject: [PATCH 5/9] Update ticdc/ticdc-open-api-v2.md --- ticdc/ticdc-open-api-v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-open-api-v2.md b/ticdc/ticdc-open-api-v2.md index c2fad7742f4f8..8c5185b602264 100644 --- a/ticdc/ticdc-open-api-v2.md +++ b/ticdc/ticdc-open-api-v2.md @@ -292,7 +292,7 @@ The `consistent` parameters are described as follows: | `use_file_backend` | `BOOL` type. Specifies whether to store the redo log in a local file. (Optional) | | `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | | `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | -| `compression` | `STRING` type. The behavior to compress redo log files, available options are "" and "lz4".. (Optional) | +| `compression` | `STRING` type. The behavior to compress redo log files. Available options are `""` and `"lz4"`. The default value is `""`, which means no compression. (Optional) | | `flush_concurrency` | `INT` type. The concurrency for uploading a single file. (Optional) | The `filter` parameters are described as follows: From 9e3f95883ee5e04dfe8720e41241ca7aea3325ff Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 7 Dec 2023 17:50:21 +0800 Subject: [PATCH 6/9] Update ticdc/ticdc-open-api-v2.md --- ticdc/ticdc-open-api-v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-open-api-v2.md b/ticdc/ticdc-open-api-v2.md index 8c5185b602264..69a63e5f79999 100644 --- a/ticdc/ticdc-open-api-v2.md +++ b/ticdc/ticdc-open-api-v2.md @@ -293,7 +293,7 @@ The `consistent` parameters are described as follows: | `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | | `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | | `compression` | `STRING` type. The behavior to compress redo log files. Available options are `""` and `"lz4"`. The default value is `""`, which means no compression. (Optional) | -| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. (Optional) | +| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. The default value is `1`, which means concurrency is disabled(Optional) | The `filter` parameters are described as follows: From bd2c5c74c26a224b470c47fdbdd82ee4ad5f160f Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 7 Dec 2023 17:50:36 +0800 Subject: [PATCH 7/9] Update ticdc/ticdc-open-api-v2.md --- ticdc/ticdc-open-api-v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-open-api-v2.md b/ticdc/ticdc-open-api-v2.md index 69a63e5f79999..80626e4993741 100644 --- a/ticdc/ticdc-open-api-v2.md +++ b/ticdc/ticdc-open-api-v2.md @@ -293,7 +293,7 @@ The `consistent` parameters are described as follows: | `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | | `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | | `compression` | `STRING` type. The behavior to compress redo log files. Available options are `""` and `"lz4"`. The default value is `""`, which means no compression. (Optional) | -| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. The default value is `1`, which means concurrency is disabled(Optional) | +| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. The default value is `1`, which means concurrency is disabled (Optional) | The `filter` parameters are described as follows: From caa84c31ca427f4859cd3f71fce15e8d8c13b5bf Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 8 Dec 2023 11:10:41 +0800 Subject: [PATCH 8/9] Update ticdc/ticdc-open-api-v2.md --- ticdc/ticdc-open-api-v2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-open-api-v2.md b/ticdc/ticdc-open-api-v2.md index 80626e4993741..08c640d446160 100644 --- a/ticdc/ticdc-open-api-v2.md +++ b/ticdc/ticdc-open-api-v2.md @@ -293,7 +293,7 @@ The `consistent` parameters are described as follows: | `encoding_worker_num` | `INT` type. The number of encoding and decoding workers in the redo module. (Optional) | | `flush_worker_num` | `INT` type. The number of flushing workers in the redo module. (Optional) | | `compression` | `STRING` type. The behavior to compress redo log files. Available options are `""` and `"lz4"`. The default value is `""`, which means no compression. (Optional) | -| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. The default value is `1`, which means concurrency is disabled (Optional) | +| `flush_concurrency` | `INT` type. The concurrency for uploading a single file. The default value is `1`, which means concurrency is disabled. (Optional) | The `filter` parameters are described as follows: From 2bcc9b7d066447754c194de8e903a7c39b37ea6f Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 8 Dec 2023 11:28:11 +0800 Subject: [PATCH 9/9] Update ticdc-sink-to-kafka.md --- ticdc/ticdc-sink-to-kafka.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-sink-to-kafka.md b/ticdc/ticdc-sink-to-kafka.md index 0a41bfc528597..e5bb7ef7f1844 100644 --- a/ticdc/ticdc-sink-to-kafka.md +++ b/ticdc/ticdc-sink-to-kafka.md @@ -134,7 +134,7 @@ The following are examples when using Kafka SASL authentication: The minimum set of permissions required for TiCDC to function properly is as follows. - - The `Create` and `Write` permissions for the Topic [resource type](https://docs.confluent.io/platform/current/kafka/authorization.html#resources). + - The `Create`, `Write`, and `Describe` permissions for the Topic [resource type](https://docs.confluent.io/platform/current/kafka/authorization.html#resources). - The `DescribeConfigs` permission for the Cluster resource type. ### Integrate TiCDC with Kafka Connect (Confluent Platform)