Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unexpected cpu usage caused by tidb_server_memory_gc_trigger #48741

Closed
XuHuaiyu opened this issue Nov 21, 2023 · 0 comments · Fixed by #48927
Closed

unexpected cpu usage caused by tidb_server_memory_gc_trigger #48741

XuHuaiyu opened this issue Nov 21, 2023 · 0 comments · Fixed by #48927

Comments

@XuHuaiyu
Copy link
Contributor

XuHuaiyu commented Nov 21, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. set global tidb_server_memory_limit = "760MB"
  2. set global tidb_server_memory_limit_sess_min_size=107374182400
  3. modify the tidb code to forbid pointget
diff --git a/planner/optimize.go b/planner/optimize.go
index b0c5b0070d..76da86e391 100644
--- a/planner/optimize.go
+++ b/planner/optimize.go
@@ -211,6 +211,7 @@ func Optimize(ctx context.Context, sctx sessionctx.Context, node ast.Node, is in
                        fp = fpv.Plan
                } else {
                        fp = core.TryFastPlan(sctx, node)
+                       fp = nil
                }
                if fp != nil {
                        return fp, fp.OutputNames(), nil
  1. import tpcc warehouse 20 data
  2. tiup bench tpcc run --db tpcc20wh --threads 16 --host --port --statusPort
  3. check the grafana, we can see the cpu usage is high as the what I see instead part of following image show
  4. modify the tidb code , and we can see the cpu usage is normal as the expect to see part of following image show
diff --git a/util/gctuner/memory_limit_tuner.go b/util/gctuner/memory_limit_tuner.go
index 204c569c0c..0e7d0d3ec4 100644
--- a/util/gctuner/memory_limit_tuner.go
+++ b/util/gctuner/memory_limit_tuner.go
@@ -128,6 +128,9 @@ func (t *memoryLimitTuner) GetPercentage() float64 {
 // UpdateMemoryLimit updates the memory limit.
 // This function should be called when `tidb_server_memory_limit` or `tidb_server_memory_limit_gc_trigger` is modified.
 func (t *memoryLimitTuner) UpdateMemoryLimit() {
+       if t.waitingReset.Load() {
+               return
+       }
        var memoryLimit = t.calcMemoryLimit(t.GetPercentage())
        if memoryLimit == math.MaxInt64 {
                t.isTuning.Store(false)

image

Analysis:

  1. The tidb-server utilizes tidb_server_memory_limit_gc_trigger to set the memory threshold that actively triggers the golang garbage collection (GC). When the heapInUse memory reaches the threshold of tidb_server_memory_limit * tidb_server_memory_limit_gc_trigger, it actively triggers a golang GC. To avoid overly frequent GC, in the code implementation, after this threshold is touched once, it is adjusted from the default value of 0.7 to 1.1, and after 1 minute, it is reset back to 0.7.

  2. Due to the periodic update of global system variables by tidb-server, the operation of increasing tidb_server_memory_limit_gc_trigger from 0.7 to 1.1 to reduce the GC frequency is almost ineffective. Therefore, when the heapInUse of tidb-server remains above the threshold of tidb_server_memory_limit * tidb_server_memory_limit_gc_trigger for an extended period, a high-frequency GC phenomenon occurs, leading to the CPU being in a near-maximum utilization state for an extended duration.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

v7.3.0

@XuHuaiyu XuHuaiyu added the type/bug This issue is a bug. label Nov 21, 2023
@ti-chi-bot ti-chi-bot bot added may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Nov 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants