util: record more information before OOM #37987

mengxin9014 · 2022-09-20T08:13:24Z

What problem does this PR solve?

Issue Number: close ##37409

Problem Summary:

What is changed and how it works?

record more information：

topN-mem-consumer

current explain analyze
desensitized SQL text

memory related variables

system variables
session variables

modify record logic:

we will record the information if (current_mem_usage_ratio > 70% && (current_time - last_record_time > 60s || current_mem_usage_ratio - last_record_mem_usage_ratio > 10%))

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

record more information before OOM.

ti-chi-bot · 2022-09-20T08:13:26Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

XuHuaiyu
wshwsh12

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

mengxin9014 · 2022-09-20T09:08:35Z

/cc @XuHuaiyu

mengxin9014 · 2022-09-20T09:08:49Z

/cc @wshwsh12

…n9014/tidb into mx/recordMoreInformationBeforeOOM

XuHuaiyu · 2022-10-09T07:14:38Z

util/memoryusagealarm/memoryusagealarm.go

+)
+
+func (reason AlarmReason) String() string {
+	return [...]string{"memory increase too fast", "memory used exceed", "no reason"}[reason]


Suggested change

return [...]string{"memory increase too fast", "memory used exceed", "no reason"}[reason]

return [...]string{"memory usage grows too fast", "memory usage exceeds alarm ratio", "no reason"}[reason]

XuHuaiyu · 2022-10-09T07:18:19Z

util/memoryusagealarm/memoryusagealarm.go

+const (
+	// IncreaseFast is the reason that memory increasing too fast.
+	IncreaseFast AlarmReason = iota
+	// MemoryExceed is the reason that memory used exceed threshold.
+	MemoryExceed
+	// NoReason means no alarm
+	NoReason
+)


Suggested change

const (

// IncreaseFast is the reason that memory increasing too fast.

IncreaseFast AlarmReason = iota

// MemoryExceed is the reason that memory used exceed threshold.

MemoryExceed

// NoReason means no alarm

NoReason

)

const (

// GrowTooFast is the reason that memory usage grows too fast.

GrowTooFast AlarmReason = iota

// ExceedAlarmRatio is the reason that memory usage exceeds the alarm ratio.

ExceedAlarmRatio

// NoReason means no alarm

NoReason

)

XuHuaiyu · 2022-10-09T07:31:28Z

util/memoryusagealarm/memoryusagealarm.go

+		record.lastCheckTime = time.Now()
+		record.lastRecordMemUsed = memoryUsage
+		record.doRecord(memoryUsage, instanceStats.HeapAlloc, sm, reason)
+		record.tryRemoveNoNeedRecords()


tryRemoveRedundantRecords

tryRemoveRedundantRecords

done.

XuHuaiyu · 2022-10-09T09:58:22Z

config/config_test.go

 # The alarm threshold when memory usage of the tidb-server exceeds. The valid value range is greater than or equal to 0
-# and less than or equal to 1. The default value is 0.8.
+# and less than or equal to 1. The default value is 0.7.
 # If this configuration is set to 0 or 1, it'll disable the alarm.
 # <snip>
-memory-usage-alarm-ratio = 0.8
+memory-usage-alarm-ratio = 0.7


remove this block

XuHuaiyu · 2022-10-09T10:00:29Z

session/session_test/session_test.go

+	tk.MustQuery("select @@global.tidb_memory_usage_alarm_keep_record_num").Check(testkit.Rows("100"))
+	tk.MustExec("set @@global.tidb_memory_usage_alarm_keep_record_num=0")
+	tk.MustQuery("SHOW WARNINGS").Check(testkit.Rows("Warning 1292 Truncated incorrect tidb_memory_usage_alarm_keep_record_num value: '0'"))
+	tk.MustQuery("select @@global.tidb_memory_usage_alarm_keep_record_num").Check(testkit.Rows("1"))


Add a test for the MAXVALUE

Add a test for the MAXVALUE

done.

…n9014/tidb into mx/recordMoreInformationBeforeOOM

wshwsh12

LGTM

XuHuaiyu · 2022-10-12T02:19:59Z

/merge

ti-chi-bot · 2022-10-12T02:20:03Z

This pull request has been accepted and is ready to merge.

Commit hash: 2618c09

ti-chi-bot · 2022-10-12T02:20:15Z

@mengxin9014: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

record more information before OOM

a9bf1e0

mengxin9014 requested a review from a team as a code owner September 20, 2022 08:13

ti-chi-bot added release-note size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 20, 2022

mengxin9014 added 3 commits September 20, 2022 16:25

change variable name

0d135f5

modify tests

4a8330d

modify tests

6342b71

ti-chi-bot requested a review from XuHuaiyu September 20, 2022 09:08

ti-chi-bot requested a review from wshwsh12 September 20, 2022 09:08

auto update viriable

c707455

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 20, 2022

mengxin9014 and others added 11 commits September 20, 2022 18:56

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

c997191

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

13d7fe5

refactor memory usage alarm

4da2738

Merge branch 'mx/recordMoreInformationBeforeOOM' of github.com:mengxi…

d750a08

…n9014/tidb into mx/recordMoreInformationBeforeOOM

add tests

5284722

fmt

4895b06

fmt

526e1bc

add license header

76bf701

fmt

de38768

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

e390229

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

d0cf8ca

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 25, 2022

mengxin9014 added 2 commits September 26, 2022 18:02

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

e5d46f8

Merge branch 'mx/recordMoreInformationBeforeOOM' of github.com:mengxi…

5fc5865

…n9014/tidb into mx/recordMoreInformationBeforeOOM

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 26, 2022

XuHuaiyu reviewed Oct 9, 2022

View reviewed changes

mengxin9014 added 5 commits October 9, 2022 15:43

deprecate instance tidb memory use ratio

62427a0

modify according comments

92f9d10

modify according comments

a2ebbb3

change struct name

2c09f15

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

78e5cfd

XuHuaiyu reviewed Oct 9, 2022

View reviewed changes

XuHuaiyu added the type/enhancement label Oct 9, 2022

mengxin9014 added 2 commits October 9, 2022 18:37

add tests

98fa4f3

Merge branch 'mx/recordMoreInformationBeforeOOM' of github.com:mengxi…

2d6d81c

…n9014/tidb into mx/recordMoreInformationBeforeOOM

XuHuaiyu approved these changes Oct 10, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 10, 2022

mengxin9014 added 5 commits October 10, 2022 11:26

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

491ab76

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

dc32dc6

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

fdc6d37

fix tests

3d34222

Merge branch 'mx/recordMoreInformationBeforeOOM' of github.com:mengxi…

2618c09

…n9014/tidb into mx/recordMoreInformationBeforeOOM

wshwsh12 approved these changes Oct 11, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 11, 2022

mengxin9014 mentioned this pull request Oct 11, 2022

add doc for record OOM alarm pingcap/docs-cn#11577

Merged

14 tasks

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 12, 2022

Merge branch 'master' into mx/recordMoreInformationBeforeOOM

3141a1e

ti-chi-bot merged commit d10e195 into pingcap:master Oct 12, 2022

hi-rustin mentioned this pull request Oct 17, 2022

oom_record can not be read on GitHub Actions #38494

Closed

mengxin9014 deleted the mx/recordMoreInformationBeforeOOM branch November 12, 2022 10:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

util: record more information before OOM #37987

util: record more information before OOM #37987

mengxin9014 commented Sep 20, 2022 •

edited

ti-chi-bot commented Sep 20, 2022 •

edited

mengxin9014 commented Sep 20, 2022

mengxin9014 commented Sep 20, 2022

XuHuaiyu Oct 9, 2022

mengxin9014 Oct 9, 2022

XuHuaiyu Oct 9, 2022

mengxin9014 Oct 9, 2022

XuHuaiyu Oct 9, 2022

mengxin9014 Oct 9, 2022

XuHuaiyu Oct 9, 2022

XuHuaiyu Oct 9, 2022

mengxin9014 Oct 9, 2022

wshwsh12 left a comment

XuHuaiyu commented Oct 12, 2022

ti-chi-bot commented Oct 12, 2022

ti-chi-bot commented Oct 12, 2022

	return [...]string{"memory increase too fast", "memory used exceed", "no reason"}[reason]
	return [...]string{"memory usage grows too fast", "memory usage exceeds alarm ratio", "no reason"}[reason]

util: record more information before OOM #37987

util: record more information before OOM #37987

Conversation

mengxin9014 commented Sep 20, 2022 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Sep 20, 2022 • edited

mengxin9014 commented Sep 20, 2022

mengxin9014 commented Sep 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wshwsh12 left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Oct 12, 2022

ti-chi-bot commented Oct 12, 2022

ti-chi-bot commented Oct 12, 2022

mengxin9014 commented Sep 20, 2022 •

edited

ti-chi-bot commented Sep 20, 2022 •

edited