Potential memory leak on TiDB server

[coderplay]

Bug Report

Please answer these questions before submitting your issue. Thanks!

I don't know how to reproduce it. We notice there is one TiDB server in our cluster with 7.9 GB memory usage, the rest TiDB servers are empty, and we haven't run queries on that cluster for quite a while. The show processlist returns nothing except the session we run this show command.

Here is the heap dump (click the image):

Any clues on that?

[XuHuaiyu]

The memory usage is consumed by the internal background worker.
PTAL @chrysan

[chrysan]

@zeminzhou what's your tidb_version and analyze_version?

[coderplay]

what's your tidb_version and analyze_version?

tidb_version is 5.2.2, how can I get the analyze_version ?

Is that a dangling analyze?

[chrysan]

what's your tidb_version and analyze_version?

tidb_version is 5.2.2, how can I get the analyze_version ?

Is that a dangling analyze?

show global variables like "%tidb_analyze_version%";

[chrysan]

We enhanced the memory usage of analyze in v5.3 (with tidb_analyze_version=2 which is the default set). Please try with it, if you have to stick to v5.2.2, please set @@tidb_analyze_version=1 referring to this KB https://kb.pingcap.com/post/solutions/405 as a workaround.

[coderplay]

Could you kindly educate me on how the analyze version caused the memory leakage?

[coderplay]

@XuHuaiyu

The memory usage is consumed by the internal background worker.

Are you meaning this is a fixed memory cost for TiDB servers because it's a long-lived background worker? That doesn't explain why the other TiDB servers on the cluster don't have this part of memory footprints.

[chrysan]

@coderplay the major memory cost is from analyze job, which is a scheduled background process, instead of long-living. Background analyze job is scheduled mainly based on 1. the ratio of modification on this table 2. if current time is within the window set.

For tidb-version < v5.1.0, the default analyze-version is 1.
From tidb-version v5.1.0, to enhance the statistics' accuracy, we provide analyze-version 2, while the new sampling algorithm consumes more memory at TiDB server side.
From tidb-version v5.3.0, we enhanced the sampling algorithm to reduce the TiDB side memory usage.
That's the reason of my above suggestion.

[coderplay]

@chrysan Should be related to this: #29306

[chrysan]

@chrysan Should be related to this: #29306

Yes, you are right. I missed it. If you see memory usage always high, it could be the memory leak issue. If the memory usage is sometimes high but sometimes drops, it could be the limitation of the old sampling algorithm.

Background auto-analyze jobs can be running but not shown in the results of show processlist. We are working to enhance this part.

You can upgrade to v5.3 to get stable analyze v2 or just workaround to use analyze v1.

[coderplay]

Thanks for the confirmation. Since it's fixed, I will go ahead and close this issue.

[tiancaiamao]

Duplicated by #32499