Skip to content

Commit

Permalink
Add docs for global memory control GA (#11419)
Browse files Browse the repository at this point in the history
  • Loading branch information
TomShawn committed Dec 6, 2022
1 parent a6e8361 commit 8c456f0
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 48 deletions.
31 changes: 4 additions & 27 deletions configure-memory-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,25 +38,7 @@ SET tidb_mem_quota_query = 8 << 10;

## Configure the memory usage threshold of a tidb-server instance

In the TiDB configuration file, you can set the memory usage threshold of a tidb-server instance by configuring [`server-memory-quota`](/tidb-configuration-file.md#server-memory-quota-new-in-v409).

The following example sets the total memory usage of a tidb-server instance to 32 GB:

{{< copyable "" >}}

```toml
[performance]
server-memory-quota = 34359738368
```

In this configuration, when the memory usage of a tidb-server instance reaches 32 GB, the instance starts to kill running SQL statements randomly until the memory usage drops below 32 GB. SQL operations that are forced to terminate return an `Out Of Global Memory Limit!` error message to the client.

> **Warning:**
>
> + `server-memory-quota` is still an experimental feature. It is **NOT** recommended that you use it in a production environment.
> + The default value of `server-memory-quota` is `0`, which means no memory limit.
Since v6.4.0, you can use the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) to set the threshold for the memory usage of a tidb-server instance.
Since v6.5.0, you can use the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) to set the threshold for the memory usage of a tidb-server instance.

For example, set the total memory usage of a tidb-server instance to 32 GB:

Expand All @@ -74,19 +56,14 @@ Currently, the memory limit set by `tidb_server_memory_limit` **DOES NOT** termi

> **Warning:**
>
> + The global memory control of tidb-server instances is still an experimental feature. It is not recommended to use it in the production environment.
> + During the startup process, TiDB does not guarantee that the [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) limit is enforced. If the free memory of the operating system is insufficient, TiDB might still encounter OOM. You need to ensure that the TiDB instance has enough available memory.
> + In the process of memory control, the total memory usage of TiDB might slightly exceed the limit set by `tidb_server_memory_limit`.
> + To ensure compatibility, when `tidb_server_memory_limit` is enabled, the system ignores the `server-memory-quota` value and uses `tidb_server_memory_limit` memory control mechanism. When `tidb_server_memory_limit` is disabled, the system uses the `server-memory-quota` value to control the memory usage of the tidb-server instance.
> + Since v6.5.0, the configruation item `server-memory-quota` is deprecated. To ensure compatibility, after you upgrade your cluster to v6.5.0 or a later version, `tidb_server_memory_limit` will inherit the value of `server-memory-quota`. If you have not configured `server-memory-quota` before the upgrade, the default value of `tidb_server_memory_limit` is used, which is `80%`.
When the memory usage of a tidb-server instance reaches a certain proportion of the total memory (the proportion is controlled by the system variable [`tidb_server_memory_limit_gc_trigger`](/system-variables.md#tidb_server_memory_limit_gc_trigger-new-in-v640)), tidb-server will try to trigger a Golang GC to relieve memory stress. To avoid frequent GCs that cause performance issues due to the instance memory fluctuating around the threshold, this GC method will trigger GC at most once every minute.

## View the memory usage of the current tidb-server instance using the INFORMATION_SCHEMA system table

> **Warning:**
>
> The following system tables are introduced in v6.4.0. Currently, these tables are still experimental. The memory usage information provided is only for reference. It is not recommended to use the following system tables in a production environment to obtain memory usage information for decision-making.
To view the memory usage of the current instance or cluster, you can query the system table [`INFORMATION_SCHEMA.(CLUSTER_)MEMORY_USAGE`](/information-schema/information-schema-memory-usage.md).

To view the memory-related operations and execution basis of the current instance or cluster, you can query the system table [`INFORMATION_SCHEMA.(CLUSTER_)MEMORY_USAGE_OPS_HISTORY`](/information-schema/information-schema-memory-usage-ops-history.md). For each instance, this table retains the latest 50 records.
Expand Down Expand Up @@ -136,12 +113,12 @@ The following example constructs a memory-intensive SQL statement that triggers
4. Check the `tidb.log` file which records the total system memory, current system memory usage, memory usage of the tidb-server instance, and the directory of status files.

```
[2022/10/11 16:39:02.281 +08:00] [WARN] [memoryusagealarm.go:212] ["tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path"] ["is server-memory-quota set"=false] ["system memory total"=33682427904] ["system memory usage"=22120655360] ["tidb-server memory usage"=21468556992] [memory-usage-alarm-ratio=0.85] ["record path"=/tiup/deploy/tidb-4000/log/oom_record]
[2022/10/11 16:39:02.281 +08:00] [WARN] [memoryusagealarm.go:212] ["tidb-server has the risk of OOM because of memory usage exceeds alarm ratio. Running SQLs and heap profile will be recorded in record path"] ["is tidb_server_memory_limit set"=false] ["system memory total"=33682427904] ["system memory usage"=22120655360] ["tidb-server memory usage"=21468556992] [memory-usage-alarm-ratio=0.85] ["record path"=/tiup/deploy/tidb-4000/log/oom_record]
```

The fields of the example log file above are described as follows:

* `is server-memory-quota set` indicates whether [`server-memory-quota`](/tidb-configuration-file.md#server-memory-quota-new-in-v409) is set.
* `is tidb_server_memory_limit set` indicates whether [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640) is set.
* `system memory total` indicates the total memory of the current system.
* `system memory usage` indicates the current system memory usage.
* `tidb-server memory usage` indicates the memory usage of the tidb-server instance.
Expand Down
1 change: 0 additions & 1 deletion experimental-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ This document introduces the experimental features of TiDB in different versions
+ [FastScan](/develop/dev-guide-use-fastscan.md). (Introduced in v6.2.0)
+ [Extended statistics](/extended-statistics.md). (Introduced in v5.0.0)
+ [Randomly sample about 10000 rows of data to quickly build statistics](/system-variables.md#tidb_enable_fast_analyze) (Introduced in v3.0)
+ [Globally control the memory usage of a tidb-server instance](/configure-memory-usage.md#configure-the-memory-usage-threshold-of-a-tidb-server-instance). (Introduced in v6.4.0)

## Stability

Expand Down
21 changes: 4 additions & 17 deletions system-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -2566,9 +2566,8 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified
- When this variable is configured to `0` or `1`, it means the memory threshold alarm feature is disabled.
- When this variable is configured to a value greater than `0` and less than `1`, it means that the memory threshold alarm feature is enabled.

- If the system variable [`tidb_server_memory_limit`](#tidb_server_memory_limit-new-in-v640) is `0` and the [`server-memory-quota`](/tidb-configuration-file.md#server-memory-quota-new-in-v409) configuration item is not set, the memory alarm threshold is `tidb_memory-usage-alarm-ratio * system memory size`.
- If the system variable `tidb_server_memory_limit` is `0` and the `server-memory-quota` configuration item is set to greater than 0, the memory alarm threshold is `tidb_memory-usage-alarm-ratio * server-memory-quota`.
- If the system variable `tidb_server_memory_limit` is set to greater than 0, the memory alarm threshold is `tidb_memory-usage-alarm-ratio * tidb_server_memory_limit`.
- If the value of the system variable [`tidb_server_memory_limit`](#tidb_server_memory_limit-new-in-v640) is `0`, the memory alarm threshold is `tidb_memory-usage-alarm-ratio * system memory size`.
- If the value of the system variable `tidb_server_memory_limit` is set to greater than 0, the memory alarm threshold is `tidb_memory-usage-alarm-ratio * tidb_server_memory_limit`.

</CustomContent>

Expand Down Expand Up @@ -3507,13 +3506,9 @@ SHOW WARNINGS;

### tidb_server_memory_limit <span class="version-mark">New in v6.4.0</span>

> **Warning:**
>
> Currently, `tidb_server_memory_limit` is still experimental. It is not recommended to use it in the production environment.
- Scope: GLOBAL
- Persists to cluster: Yes
- Default value: `0`
- Default value: `80%`
- Range:
- You can set the value in the percentage format, which means the percentage of the memory usage relative to the total memory. The value range is `[1%, 99%]`.
- You can also set the value in memory size in bytes. The value range is `[0, 9223372036854775807]`. The memory format with the units "KB|MB|GB|TB" is supported. The `0` value means no memory limit.
Expand All @@ -3523,10 +3518,6 @@ SHOW WARNINGS;

### tidb_server_memory_limit_gc_trigger <span class="version-mark">New in v6.4.0</span>

> **Warning:**
>
> Currently, `tidb_server_memory_limit_gc_trigger` is still experimental. It is not recommended to use it in the production environment.
- Scope: GLOBAL
- Persists to cluster: Yes
- Default value: `70%`
Expand All @@ -3535,14 +3526,10 @@ SHOW WARNINGS;

### tidb_server_memory_limit_sess_min_size <span class="version-mark">New in v6.4.0</span>

> **Warning:**
>
> Currently, `tidb_server_memory_limit_sess_min_size` is still experimental. It is not recommended to use it in the production environment.
- Scope: GLOBAL
- Persists to cluster: Yes
- Default value: `134217728` (which is 128 MB)
- Range: `[128, 9223372036854775807]`, in bytes
- Range: `[128, 9223372036854775807]`, in bytes. The memory format with the units "KB|MB|GB|TB" is also supported.
- After you enable the memory limit, TiDB will terminate the SQL statement with the highest memory usage on the current instance. This variable specifies the minimum memory usage of the SQL statement to be terminated. If the memory usage of a TiDB instance that exceeds the limit is caused by too many sessions with low memory usage, you can properly lower the value of this variable to allow more sessions to be canceled.

### tidb_shard_allocate_step <span class="version-mark">New in v5.0</span>
Expand Down
2 changes: 1 addition & 1 deletion tidb-configuration-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ Configuration items related to performance.

> **Warning:**
>
> `server-memory-quota` is still an experimental feature. It is **NOT** recommended that you use it in a production environment.
> Since v6.5.0, the `server-memory-quota` configuration item is deprecated and replaced by the system variable [`tidb_server_memory_limit`](/system-variables.md#tidb_server_memory_limit-new-in-v640).
+ The memory usage limit of tidb-server instances.
+ Default value: `0` (in bytes), which means no memory limit.
Expand Down
4 changes: 2 additions & 2 deletions troubleshoot-tidb-oom.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ To locate the root cause of an OOM issue, you need to collect the following info
- `mem-quota-query`
- `oom-action`
- `tidb_enable_rate_limit_action`
- `server-memory-quota`
- `tidb_server_memory_limit`
- `oom-use-tmp-storage`
- `tmp-storage-path`
- `tmp-storage-quota`
Expand All @@ -189,7 +189,7 @@ To locate the root cause of an OOM issue, you need to collect the following info
- Run `grep "tidb-server has the risk of OOM" tidb.log` to check the path of the alert file collected by TiDB Server. The following is an example output:
```shell
["tidb-server has the risk of OOM. Running SQLs and heap profile will be recorded in record path"] ["is server-memory-quota set"=false] ["system memory total"=14388137984] ["system memory usage"=11897434112] ["tidb-server memory usage"=11223572312] [memory-usage-alarm-ratio=0.8] ["record path"="/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record"]
["tidb-server has the risk of OOM. Running SQLs and heap profile will be recorded in record path"] ["is tidb_server_memory_limit set"=false] ["system memory total"=14388137984] ["system memory usage"=11897434112] ["tidb-server memory usage"=11223572312] [memory-usage-alarm-ratio=0.8] ["record path"="/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage/record"]
```
## See also
Expand Down

0 comments on commit 8c456f0

Please sign in to comment.