From 22092aa2e5df9ad1e9e447539fe547d489a7414c Mon Sep 17 00:00:00 2001 From: Ti Chi Robot Date: Thu, 4 Sep 2025 12:00:36 +0800 Subject: [PATCH 1/4] This is an automated cherry-pick of #21726 Signed-off-by: ti-chi-bot --- dynamic-config.md | 10 ++-- garbage-collection-configuration.md | 14 +++++- tikv-configuration-file.md | 76 +++++++++++++++++++++++++++++ 3 files changed, 95 insertions(+), 5 deletions(-) diff --git a/dynamic-config.md b/dynamic-config.md index 6cc16255e3b4c..60170224d22bd 100644 --- a/dynamic-config.md +++ b/dynamic-config.md @@ -136,10 +136,6 @@ The following TiKV configuration items can be modified dynamically: | `raftstore.raft-entry-cache-life-time` | The maximum remaining time allowed for the log cache in memory | | `raftstore.split-region-check-tick-interval` | The time interval at which to check whether the Region split is needed | | `raftstore.region-split-check-diff` | The maximum value by which the Region data is allowed to exceed before Region split | -| `raftstore.region-compact-check-interval` | The time interval at which to check whether it is necessary to manually trigger RocksDB compaction | -| `raftstore.region-compact-check-step` | The number of Regions checked at one time for each round of manual compaction | -| `raftstore.region-compact-min-tombstones` | The number of tombstones required to trigger RocksDB compaction | -| `raftstore.region-compact-tombstones-percent` | The proportion of tombstone required to trigger RocksDB compaction | | `raftstore.pd-heartbeat-tick-interval` | The time interval at which a Region's heartbeat to PD is triggered | | `raftstore.pd-store-heartbeat-tick-interval` | The time interval at which a store's heartbeat to PD is triggered | | `raftstore.snap-mgr-gc-tick-interval` | The time interval at which the recycle of expired snapshot files is triggered | @@ -188,6 +184,12 @@ The following TiKV configuration items can be modified dynamically: | `gc.max-write-bytes-per-sec` | The maximum bytes that can be written into RocksDB per second | | `gc.enable-compaction-filter` | Whether to enable compaction filter | | `gc.compaction-filter-skip-version-check` | Whether to skip the cluster version check of compaction filter (not released) | +| `gc.auto-compaction.check-interval` | The interval at which TiKV checks whether to trigger automatic (RocksDB) compaction | +| `gc.auto-compaction.tombstone-num-threshold` | The number of RocksDB tombstones required to trigger TiKV automatic (RocksDB) compaction | +| `gc.auto-compaction.tombstone-percent-threshold` | The percentage of RocksDB tombstones required to trigger TiKV automatic (RocksDB) compaction | +| `gc.auto-compaction.redundant-rows-threshold` | The number of redundant MVCC rows required to trigger TiKV automatic (RocksDB) compaction | +| `gc.auto-compaction.redundant-rows-percent-threshold` | The percentage of redundant MVCC rows required to trigger TiKV automatic (RocksDB) compaction | +| `gc.auto-compaction.bottommost-level-force` | Whether to force compaction on the bottommost level files in RocksDB | | `{db-name}.max-total-wal-size` | The maximum size of total WAL | | `{db-name}.max-background-jobs` | The number of background threads in RocksDB | | `{db-name}.max-background-flushes` | The maximum number of flush threads in RocksDB | diff --git a/garbage-collection-configuration.md b/garbage-collection-configuration.md index 826d5ad1c0571..d45232a377c12 100644 --- a/garbage-collection-configuration.md +++ b/garbage-collection-configuration.md @@ -106,4 +106,16 @@ show config where type = 'tikv' and name like '%enable-compaction-filter%'; | tikv | 172.16.5.36:20163 | gc.enable-compaction-filter | true | | tikv | 172.16.5.35:20163 | gc.enable-compaction-filter | true | +------+-------------------+-----------------------------+-------+ -``` \ No newline at end of file +``` + + + +> **Note:** +> +> When using the Compaction Filter mechanism, GC progress might be delayed, which can affect TiKV scan performance. If your workload contains a large number of coprocessor requests and you observe in the [**TiKV-Details > Coprocessor Detail**](/grafana-tikv-dashboard.md#coprocessor-detail) panel that the `next()` or `prev()` call count in **Total Ops Details** significantly exceeds three times the `processed_keys` calls, you can take the following actions: +> +> - For TiDB versions before v7.1.3, it is recommended to disable Compaction Filter to speed up GC. +> - For TiDB versions from v7.1.3 to v7.5.6, TiDB automatically triggers compaction based on the number of redundant versions in each Region [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and the percentage of redundant versions [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) to improve Compaction Filter GC performance. In this case, adjust these configuration items instead of disabling Compaction Filter. +> - Starting from v7.5.7, [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) are deprecated. TiDB now automatically triggers compaction based on [`gc.auto-compaction.redundant-rows-threshold`](/tikv-configuration-file.md#redundant-rows-threshold-new-in-v757) and [`gc.auto-compaction.redundant-rows-percent-threshold`](/tikv-configuration-file.md#redundant-rows-percent-threshold-new-in-v757). In this case, adjust these configuration items instead of disabling Compaction Filter. + + diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 11ad35911bffc..4f24895a945a5 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -749,12 +749,20 @@ Configuration items related to Raftstore. ### `region-compact-check-interval` +> **Warning:** +> +> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.check-interval`](#check-interval-new-in-v757). + + The time interval at which to check whether it is necessary to manually trigger RocksDB compaction. `0` means that this feature is disabled. + Default value: `"5m"` + Minimum value: `0` ### `region-compact-check-step` +> **Warning:** +> +> Starting from v7.5.7, this configuration item is deprecated. + + The number of Regions checked at one time for each round of manual compaction + Default value: @@ -764,12 +772,20 @@ Configuration items related to Raftstore. ### `region-compact-min-tombstones` +> **Warning:** +> +> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.tombstone-num-threshold`](#tombstone-num-threshold-new-in-v757). + + The number of tombstones required to trigger RocksDB compaction + Default value: `10000` + Minimum value: `0` ### `region-compact-tombstones-percent` +> **Warning:** +> +> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.tombstone-percent-threshold`](#tombstone-percent-threshold-new-in-v757). + + The proportion of tombstone required to trigger RocksDB compaction + Default value: `30` + Minimum value: `1` @@ -777,13 +793,29 @@ Configuration items related to Raftstore. ### `region-compact-min-redundant-rows` New in v7.1.0 +<<<<<<< HEAD + The number of redundant MVCC rows required to trigger RocksDB compaction. This configuration only takes effect for Partitioned Raft KV (`storage.engine="partitioned-raft-kv"`). +======= +> **Warning:** +> +> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.redundant-rows-threshold`](#redundant-rows-threshold-new-in-v757). + ++ The number of redundant MVCC rows required to trigger RocksDB compaction. +>>>>>>> 3d87baee0a (tikv: update auto compaction config (#21656) (#21726)) + Default value: `50000` + Minimum value: `0` ### `region-compact-redundant-rows-percent` New in v7.1.0 +<<<<<<< HEAD + The percentage of redundant MVCC rows required to trigger RocksDB compaction. This configuration only takes effect for Partitioned Raft KV (`storage.engine="partitioned-raft-kv"`). +======= +> **Warning:** +> +> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.redundant-rows-percent-threshold`](#redundant-rows-percent-threshold-new-in-v757). + ++ The percentage of redundant MVCC rows required to trigger RocksDB compaction. +>>>>>>> 3d87baee0a (tikv: update auto compaction config (#21656) (#21726)) + Default value: `20` + Minimum value: `1` + Maximum value: `100` @@ -2051,6 +2083,50 @@ Configuration items related to TiDB Lightning import and BR restore. + The number of GC threads when `enable-compaction-filter` is `false`. + Default value: `1` +## gc.auto-compaction + +Configures the behavior of TiKV automatic compaction. + +### `check-interval` New in v7.5.7 + ++ The interval at which TiKV checks whether to trigger automatic compaction. Within this interval, Regions that meet the automatic compaction conditions are processed based on priority. When the interval elapses, TiKV rescans Region information and recalculates priorities. ++ Default value: `"300s"` + +### `tombstone-num-threshold` New in v7.5.7 + ++ The number of RocksDB tombstones required to trigger TiKV automatic compaction. When the number of tombstones reaches this threshold, or when the percentage of tombstones reaches [`tombstone-percent-threshold`](#tombstone-percent-threshold-new-in-v757), TiKV triggers automatic compaction. ++ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is disabled. ++ Default value: `10000` ++ Minimum value: `0` + +### `tombstone-percent-threshold` New in v7.5.7 + ++ The percentage of RocksDB tombstones required to trigger TiKV automatic compaction. When the percentage of tombstones reaches this threshold, or when the number of tombstones reaches [`tombstone-num-threshold`](#tombstone-num-threshold-new-in-v757), TiKV triggers automatic compaction. ++ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is disabled. ++ Default value: `30` ++ Minimum value: `0` ++ Maximum value: `100` + +### `redundant-rows-threshold` New in v7.5.7 + ++ The number of redundant MVCC rows required to trigger TiKV automatic compaction. Redundant rows include RocksDB tombstones, TiKV stale versions, and TiKV deletion tombstones. When the number of redundant MVCC rows reaches this threshold, or when the percentage of these rows reaches [`redundant-rows-percent-threshold`](#redundant-rows-percent-threshold-new-in-v757), TiKV triggers automatic compaction. ++ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is enabled. ++ Default value: `50000` ++ Minimum value: `0` + +### `redundant-rows-percent-threshold` New in v7.5.7 + ++ The percentage of redundant MVCC rows required to trigger TiKV automatic compaction. Redundant rows include RocksDB tombstones, TiKV stale versions, and TiKV deletion tombstones. When the number of redundant MVCC rows reaches [`redundant-rows-threshold`](#redundant-rows-threshold-new-in-v757), or when the percentage of these rows reaches `redundant-rows-percent-threshold`, TiKV triggers automatic compaction. ++ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is enabled. ++ Default value: `20` ++ Minimum value: `0` ++ Maximum value: `100` + +### `bottommost-level-force` New in v7.5.7 + ++ Controls whether to force compaction on the bottommost files in RocksDB. ++ Default value: `true` + ## backup Configuration items related to BR backup. From d654e7579292930417d23ce596e02a48d1d40990 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 4 Sep 2025 12:07:32 +0800 Subject: [PATCH 2/4] Discard changes to dynamic-config.md --- dynamic-config.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/dynamic-config.md b/dynamic-config.md index 60170224d22bd..6cc16255e3b4c 100644 --- a/dynamic-config.md +++ b/dynamic-config.md @@ -136,6 +136,10 @@ The following TiKV configuration items can be modified dynamically: | `raftstore.raft-entry-cache-life-time` | The maximum remaining time allowed for the log cache in memory | | `raftstore.split-region-check-tick-interval` | The time interval at which to check whether the Region split is needed | | `raftstore.region-split-check-diff` | The maximum value by which the Region data is allowed to exceed before Region split | +| `raftstore.region-compact-check-interval` | The time interval at which to check whether it is necessary to manually trigger RocksDB compaction | +| `raftstore.region-compact-check-step` | The number of Regions checked at one time for each round of manual compaction | +| `raftstore.region-compact-min-tombstones` | The number of tombstones required to trigger RocksDB compaction | +| `raftstore.region-compact-tombstones-percent` | The proportion of tombstone required to trigger RocksDB compaction | | `raftstore.pd-heartbeat-tick-interval` | The time interval at which a Region's heartbeat to PD is triggered | | `raftstore.pd-store-heartbeat-tick-interval` | The time interval at which a store's heartbeat to PD is triggered | | `raftstore.snap-mgr-gc-tick-interval` | The time interval at which the recycle of expired snapshot files is triggered | @@ -184,12 +188,6 @@ The following TiKV configuration items can be modified dynamically: | `gc.max-write-bytes-per-sec` | The maximum bytes that can be written into RocksDB per second | | `gc.enable-compaction-filter` | Whether to enable compaction filter | | `gc.compaction-filter-skip-version-check` | Whether to skip the cluster version check of compaction filter (not released) | -| `gc.auto-compaction.check-interval` | The interval at which TiKV checks whether to trigger automatic (RocksDB) compaction | -| `gc.auto-compaction.tombstone-num-threshold` | The number of RocksDB tombstones required to trigger TiKV automatic (RocksDB) compaction | -| `gc.auto-compaction.tombstone-percent-threshold` | The percentage of RocksDB tombstones required to trigger TiKV automatic (RocksDB) compaction | -| `gc.auto-compaction.redundant-rows-threshold` | The number of redundant MVCC rows required to trigger TiKV automatic (RocksDB) compaction | -| `gc.auto-compaction.redundant-rows-percent-threshold` | The percentage of redundant MVCC rows required to trigger TiKV automatic (RocksDB) compaction | -| `gc.auto-compaction.bottommost-level-force` | Whether to force compaction on the bottommost level files in RocksDB | | `{db-name}.max-total-wal-size` | The maximum size of total WAL | | `{db-name}.max-background-jobs` | The number of background threads in RocksDB | | `{db-name}.max-background-flushes` | The maximum number of flush threads in RocksDB | From 99d2fd0c7dd3df30461532112d5fe495e918ae95 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 4 Sep 2025 12:07:40 +0800 Subject: [PATCH 3/4] Discard changes to tikv-configuration-file.md --- tikv-configuration-file.md | 76 -------------------------------------- 1 file changed, 76 deletions(-) diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 4f24895a945a5..11ad35911bffc 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -749,20 +749,12 @@ Configuration items related to Raftstore. ### `region-compact-check-interval` -> **Warning:** -> -> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.check-interval`](#check-interval-new-in-v757). - + The time interval at which to check whether it is necessary to manually trigger RocksDB compaction. `0` means that this feature is disabled. + Default value: `"5m"` + Minimum value: `0` ### `region-compact-check-step` -> **Warning:** -> -> Starting from v7.5.7, this configuration item is deprecated. - + The number of Regions checked at one time for each round of manual compaction + Default value: @@ -772,20 +764,12 @@ Configuration items related to Raftstore. ### `region-compact-min-tombstones` -> **Warning:** -> -> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.tombstone-num-threshold`](#tombstone-num-threshold-new-in-v757). - + The number of tombstones required to trigger RocksDB compaction + Default value: `10000` + Minimum value: `0` ### `region-compact-tombstones-percent` -> **Warning:** -> -> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.tombstone-percent-threshold`](#tombstone-percent-threshold-new-in-v757). - + The proportion of tombstone required to trigger RocksDB compaction + Default value: `30` + Minimum value: `1` @@ -793,29 +777,13 @@ Configuration items related to Raftstore. ### `region-compact-min-redundant-rows` New in v7.1.0 -<<<<<<< HEAD + The number of redundant MVCC rows required to trigger RocksDB compaction. This configuration only takes effect for Partitioned Raft KV (`storage.engine="partitioned-raft-kv"`). -======= -> **Warning:** -> -> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.redundant-rows-threshold`](#redundant-rows-threshold-new-in-v757). - -+ The number of redundant MVCC rows required to trigger RocksDB compaction. ->>>>>>> 3d87baee0a (tikv: update auto compaction config (#21656) (#21726)) + Default value: `50000` + Minimum value: `0` ### `region-compact-redundant-rows-percent` New in v7.1.0 -<<<<<<< HEAD + The percentage of redundant MVCC rows required to trigger RocksDB compaction. This configuration only takes effect for Partitioned Raft KV (`storage.engine="partitioned-raft-kv"`). -======= -> **Warning:** -> -> Starting from v7.5.7, this configuration item is deprecated and replaced by [`gc.auto-compaction.redundant-rows-percent-threshold`](#redundant-rows-percent-threshold-new-in-v757). - -+ The percentage of redundant MVCC rows required to trigger RocksDB compaction. ->>>>>>> 3d87baee0a (tikv: update auto compaction config (#21656) (#21726)) + Default value: `20` + Minimum value: `1` + Maximum value: `100` @@ -2083,50 +2051,6 @@ Configuration items related to TiDB Lightning import and BR restore. + The number of GC threads when `enable-compaction-filter` is `false`. + Default value: `1` -## gc.auto-compaction - -Configures the behavior of TiKV automatic compaction. - -### `check-interval` New in v7.5.7 - -+ The interval at which TiKV checks whether to trigger automatic compaction. Within this interval, Regions that meet the automatic compaction conditions are processed based on priority. When the interval elapses, TiKV rescans Region information and recalculates priorities. -+ Default value: `"300s"` - -### `tombstone-num-threshold` New in v7.5.7 - -+ The number of RocksDB tombstones required to trigger TiKV automatic compaction. When the number of tombstones reaches this threshold, or when the percentage of tombstones reaches [`tombstone-percent-threshold`](#tombstone-percent-threshold-new-in-v757), TiKV triggers automatic compaction. -+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is disabled. -+ Default value: `10000` -+ Minimum value: `0` - -### `tombstone-percent-threshold` New in v7.5.7 - -+ The percentage of RocksDB tombstones required to trigger TiKV automatic compaction. When the percentage of tombstones reaches this threshold, or when the number of tombstones reaches [`tombstone-num-threshold`](#tombstone-num-threshold-new-in-v757), TiKV triggers automatic compaction. -+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is disabled. -+ Default value: `30` -+ Minimum value: `0` -+ Maximum value: `100` - -### `redundant-rows-threshold` New in v7.5.7 - -+ The number of redundant MVCC rows required to trigger TiKV automatic compaction. Redundant rows include RocksDB tombstones, TiKV stale versions, and TiKV deletion tombstones. When the number of redundant MVCC rows reaches this threshold, or when the percentage of these rows reaches [`redundant-rows-percent-threshold`](#redundant-rows-percent-threshold-new-in-v757), TiKV triggers automatic compaction. -+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is enabled. -+ Default value: `50000` -+ Minimum value: `0` - -### `redundant-rows-percent-threshold` New in v7.5.7 - -+ The percentage of redundant MVCC rows required to trigger TiKV automatic compaction. Redundant rows include RocksDB tombstones, TiKV stale versions, and TiKV deletion tombstones. When the number of redundant MVCC rows reaches [`redundant-rows-threshold`](#redundant-rows-threshold-new-in-v757), or when the percentage of these rows reaches `redundant-rows-percent-threshold`, TiKV triggers automatic compaction. -+ This configuration item takes effect only when [Compaction Filter](/garbage-collection-configuration.md) is enabled. -+ Default value: `20` -+ Minimum value: `0` -+ Maximum value: `100` - -### `bottommost-level-force` New in v7.5.7 - -+ Controls whether to force compaction on the bottommost files in RocksDB. -+ Default value: `true` - ## backup Configuration items related to BR backup. From 7453f8f42102dc9f520f260107a9bb1723c410e7 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 4 Sep 2025 12:09:08 +0800 Subject: [PATCH 4/4] fix --- garbage-collection-configuration.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/garbage-collection-configuration.md b/garbage-collection-configuration.md index d45232a377c12..57af92912606f 100644 --- a/garbage-collection-configuration.md +++ b/garbage-collection-configuration.md @@ -115,7 +115,6 @@ show config where type = 'tikv' and name like '%enable-compaction-filter%'; > When using the Compaction Filter mechanism, GC progress might be delayed, which can affect TiKV scan performance. If your workload contains a large number of coprocessor requests and you observe in the [**TiKV-Details > Coprocessor Detail**](/grafana-tikv-dashboard.md#coprocessor-detail) panel that the `next()` or `prev()` call count in **Total Ops Details** significantly exceeds three times the `processed_keys` calls, you can take the following actions: > > - For TiDB versions before v7.1.3, it is recommended to disable Compaction Filter to speed up GC. -> - For TiDB versions from v7.1.3 to v7.5.6, TiDB automatically triggers compaction based on the number of redundant versions in each Region [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and the percentage of redundant versions [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) to improve Compaction Filter GC performance. In this case, adjust these configuration items instead of disabling Compaction Filter. -> - Starting from v7.5.7, [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) are deprecated. TiDB now automatically triggers compaction based on [`gc.auto-compaction.redundant-rows-threshold`](/tikv-configuration-file.md#redundant-rows-threshold-new-in-v757) and [`gc.auto-compaction.redundant-rows-percent-threshold`](/tikv-configuration-file.md#redundant-rows-percent-threshold-new-in-v757). In this case, adjust these configuration items instead of disabling Compaction Filter. +> - Starting from v7.1.3, TiDB automatically triggers compaction based on the number of redundant versions in each Region [`region-compact-min-redundant-rows`](/tikv-configuration-file.md#region-compact-min-redundant-rows-new-in-v710) and the percentage of redundant versions [`region-compact-redundant-rows-percent`](/tikv-configuration-file.md#region-compact-redundant-rows-percent-new-in-v710) to improve Compaction Filter GC performance. In this case, adjust these configuration items instead of disabling Compaction Filter.