server: close the temporary session in HTTP API to avoid memory leak #24339

tiancaiamao · 2021-04-28T04:27:46Z

What problem does this PR solve?

In one of our customer we find their TiDB memory usage is abnormal:

And the inused object is SessionStatsCollector which is allocated from the HTTP API:

Problem Summary:

What is changed and how it works?

The SessionStatsCollector is used by each session to collect its statistics changes, and merge the change with feedback and dump to TiKV.

All stats collector are linked together in the statistics handler which is a global object. As it's referenced, Go GC can not release them and that result in the memory leak.

What's Changed:

Do not allocate the SessionStatsCollector for the session object in HTTP handler.

How it Works:

Related changes

Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Release note

Handle a potential statistic object's memory leak when HTTP api is used

…mory leak

eurekaka · 2021-04-28T08:50:49Z

server/http_handler.go

+	// Disable stats collector.
+	// Stats collector of all the sessions are linked together in the statistics handler which is a global object.
+	// If we create stats collector here, they may not be recycled by the Go GC and cause memory leak.
+	session, err := session.CreateSessionWithOpt(t.Store, &session.Opt{DisableStatsCollector: true})


Seems like the root cause of the memory leak is that there is no session.Close() called for this session, so s.statsCollector.Delete() is not triggered to release the stats collector. Would there be other potential leaks caused by not closing this session?

Would there be other potential leaks caused by not closing this session?

I'm not sure. I wonder is it really necessary to create a temporary session here? Maybe that's a better fix, what's your opinion?

Agree with @eurekaka .
Calling session.Close() when exiting from these HTTP APIs seems a safer and clearer way to solve this issue.
If we add another global resource into session like statsCollector, then this problem will occur again if using this way.

qw4990

Seems that CreateSession in statistic_handler.go, sql_info_fetcher.go and gc_worker.go should also be changed?

tiancaiamao · 2021-04-29T08:56:09Z

Seems that CreateSession in statistic_handler.go, sql_info_fetcher.go and gc_worker.go should also be changed?

I'll update this PR to avoid using the temporary session... hold on

tiancaiamao · 2021-05-12T14:36:47Z

Updated, avoid unnecessary session creation and ensure session.Close is called.
PTAL @eurekaka @qw4990

eurekaka · 2021-05-13T02:59:07Z

/lgtm

ti-chi-bot · 2021-05-13T02:59:10Z

@eurekaka: /lgtm is only allowed for the reviewers in list.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

eurekaka · 2021-05-13T02:59:37Z

/lgtm

ti-chi-bot · 2021-05-13T06:07:19Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

eurekaka
qw4990

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

qw4990 · 2021-05-13T06:07:32Z

/merge

ti-chi-bot · 2021-05-13T06:07:35Z

This pull request has been accepted and is ready to merge.

Commit hash: cf7b6d4

ti-chi-bot · 2021-05-13T08:36:47Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

ti-chi-bot · 2021-05-13T09:38:48Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-05-14T01:57:22Z

cherry pick to release-5.0 in PR #24649

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-05-14T01:58:16Z

cherry pick to release-4.0 in PR #24650

…ingcap#24052) * *: fix revoke statement for CURRENT_USER() and refine error message planner: support set tidb_allow_mpp to `2` or `ENFORCE` to enforce use mpp mode. (pingcap#24516) store/tikv: remove use of SchemaAmender option in store/tikv (pingcap#24408) *: the value of tikv-client.store-liveness-timeout should not less than 0 (pingcap#24244) store/tikv: remove use of EnableAsyncCommit option in store/tikv (pingcap#24462) txn: Add txn state's view (pingcap#22908) planner: ignore lock for temporary table of PointGet and BatchPointGet (pingcap#24540) store/tikv: remove use of ReplicaRead transaction option in store/tikv (pingcap#24409) store/driver: move error to single package (pingcap#24549) ddl: add check table compatibility for temporary table (pingcap#24501) store/tikv: remove use of IsStatenessReadOnly option in store/tikv (pingcap#24464) store/tikv: change backoff type for missed tiflash peer. (pingcap#24577) store/tikv: remove use of MatchStoreLabels transaction option in store/tikv (pingcap#24465) executor, meta: Allocate auto id for global temporary tables (pingcap#24506) store/tikv: remove use of SampleStep option in store/tikv (pingcap#24461) executor: add partition pruning tests for adding and dropping partition operations (pingcap#24573) ddl: forbid partition on temporary mode before put into queue (pingcap#24565) ddl: speedup test case TestIndexOnMultipleGeneratedColumn (pingcap#24487) execution: Fix issue 24439 Inconsistent error with MySQL for GRANT CREATE USER ON <specific db>.* (pingcap#24485) *: fix errcheck (pingcap#24463) test: make TestExtractStartTs stable (pingcap#24585) ddl: forbid recover/flashback temporary tables (pingcap#24518) executor: fix point_get result on clustered index when new-row-format disabled but new-collation enabled (pingcap#24544) executor: Improve the performance of appending not fixed columns (pingcap#20969) *: typo fix (pingcap#24564) planner/core: refresh stale regions in cache for batch cop response (pingcap#24457) binlog: DML on temporary tables do not write binlog (pingcap#24570) store/tikv: remove use of GuaranteeLinearizability option in store/tikv (pingcap#24605) store/tikv: remove use of CollectRuntimeStats option in store/tikv (pingcap#24604) store/tikv: move Backoffer into a single package (pingcap#24525) variables: init cte max recursive deeps in a new session (pingcap#24609) store/tikv: move transaction options out to /kv (pingcap#24619) store/driver: move backoff driver into single package so we can use i… (pingcap#24624) server: close the temporary session in HTTP API to avoid memory leak (pingcap#24339) store/tikv: use latest PD TS plus one as min commit ts (pingcap#24579) planner: fix incorrect TableDual plan built from nulleq (pingcap#24596) ranger: fix the case which could have duplicate ranges (pingcap#24590) executor, store: Pass the SQL digest down to pessimistic lock request (pingcap#24380) kv: remove UnionStore interface (pingcap#24625) *: enable gosimple linter (pingcap#24617) txn: avoid the gc resolving pessimistic locks of ongoing transactions (pingcap#24601) util: fix wrong enum building for index range (pingcap#24632) sessionctx: change innodb large prefix default (pingcap#24555) store: fix data race about KVStore.tikvClient (pingcap#24655) executor, privileges: Add dynamic privileges to SHOW PRIVILEGES (pingcap#24646) ddl: refactor rule [4/6] (pingcap#24007) cmd: ddl_test modify retryCnt from 5 to 20 (pingcap#24662) executor: add correctness tests about direct reading with ORDER BY and LIMIT (pingcap#24455) store/tikv: remove options from unionstore (pingcap#24629) planner: fix wrongly check for update statement (pingcap#24614) store/tikv: remove CompareTS (pingcap#24657) planner, privilege: Add security enhanced mode part 4 (pingcap#24416) executor: add some test cases about partition table dynamic-mode with split-region (pingcap#24665) planner: fix wrong column offsets when processing dynamic pruning for IndexJoin (pingcap#24659) *: Add security enhanced mode part 3 (pingcap#24412) store/tikv: resolve ReplicaReadType dependencies (pingcap#24653) executor: add test cases about partition table with `expression` (pingcap#24628) tablecodec: fix write wrong prefix index value when collation is ascii_bin/latin1_bin (pingcap#24578) *: compatibility with staleread (pingcap#24285) session: test that temporary tables will also be retried (pingcap#24505) domain, session: Add new sysvarcache to replace global values cache (pingcap#24359) ddl, transaction: DDL on temporary tables won't affect transactions (pingcap#24534) *: implement tidb_bounded_staleness built-in function (pingcap#24328) executor: add correctness tests for partition table with different joins (pingcap#24673) expression: fix the spelling of word arithmetical (pingcap#24713) store/copr: balance region for batch cop task (pingcap#24521) store, metrics: Add metrics for safetTS updating (pingcap#24687) sem: add tidbredact log to restricted variables (pingcap#24701) session: fix dml_batch_size doesn't load the global variable (pingcap#24710) store/tikv: retry TSO RPC (pingcap#24682) expression, planner: push cast down to control function with enum type. (pingcap#24542) executor: add correctness tests about IndexMerge (pingcap#24674) variable: change default for DefDMLBatchSize tidbOptInt64 call (pingcap#24697) planner: add partitioning pruning tests for range partitioning (pingcap#24554) *: add option for enum push down (pingcap#24685) txn: break dependency from store/tikv to tidb/kv cause by TransactionOption (pingcap#24656) executor: enhancement for ListInDisk(support writing after reading) (pingcap#24379) kv: move TxnScope into kv (pingcap#24715) execution: fix the incorrect use of cached plan for point get (pingcap#24749) executor: add correctness tests about direct reading with indexJoin (pingcap#24497) variable: fix sysvar datarace with deep copy (pingcap#24732) *: Implementing RENAME USER (pingcap#24413) infoschema, executor: Add the deadlock table (pingcap#24524) docs: Some proposal for renaming and configurations for Lock View (pingcap#24718) planner: add range partition boundaries tests with BETWEEN expression (pingcap#24598) oracle: simplify timestamp utilities (pingcap#24688) executor: fix wrong enum key in point get (pingcap#24618) ranger: fix incorrect enum range for xxx_ci collation (pingcap#24661) executor: add some test cases about dynamic-mode and apply operator (pingcap#24683) store/tikv: remove Variables.Hook (pingcap#24758) ddl: speed up the execution time of `TestBackwardCompatibility`. (pingcap#24704) *: prepare errors for CTE (pingcap#24763) expression: support cast real/int as real (pingcap#24670) executor: add table name in log (pingcap#24666) expression: add builtin function ``json_pretty`` (pingcap#24675) ddl: make `TestDropLastVisibleColumns` stable (pingcap#24790) * ddl: make `TestDropLastVisibleColumns` stable *: support AS OF TIMESTAMP read-only begin statement (pingcap#24740) executor: Fix unstable TestTiDBLastTxnInfoCommitMode (pingcap#24779) planner: add tests for partition range boundaries for LT/GT (pingcap#24574) test: record random seed in TestIssue20658 (pingcap#24782) store/tikv/retry: define Config instead of BackoffType (pingcap#24692) config: ignore tiflash when show config (pingcap#24770) privileges: improve dynamic privs registration and tests (pingcap#24773) README: remove the link of TiDB Monthly Update (pingcap#24791) region_cache: filter peers on tombstone or dropped stores (pingcap#24726) util/stmtsummary: remove import package tikv (pingcap#24776) ddl: grammar check for create unsupported temporary table (pingcap#24723) *: update go.etcd.io/bbolt (pingcap#24799) ddl: speed up the execution time of `ddl test` and `Test Chunk pingcap#7 ddl-other` (pingcap#24780) executor: remove the unnecessary use of fmt.Sprintf (pingcap#24815) executor: fix index join panic on prefix index on some cases (pingcap#24568)

…24339) (#24650)

…24339) (#24649)

server,session: do not create stats collector in HTTP api to avoid me…

51708c7

…mory leak

tiancaiamao requested a review from eurekaka April 28, 2021 04:27

ti-chi-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 28, 2021

github-actions bot added the sig/sql-infra SIG: SQL Infra label Apr 28, 2021

fix CI

9191316

tiancaiamao changed the title ~~server,session: do not create stats collector in HTTP api to avoid memory leak~~ server,session: do not create stats collector in HTTP API to avoid memory leak Apr 28, 2021

qw4990 requested a review from rebelice April 28, 2021 08:46

eurekaka reviewed Apr 28, 2021

View reviewed changes

qw4990 self-requested a review April 29, 2021 03:09

qw4990 reviewed Apr 29, 2021

View reviewed changes

tiancaiamao added 2 commits May 12, 2021 22:28

address comment

3629b06

Merge branch 'master' into stats-leak

0d757d0

ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 12, 2021

tiancaiamao changed the title ~~server,session: do not create stats collector in HTTP API to avoid memory leak~~ server: close the temporary session in HTTP API to avoid memory leak May 13, 2021

eurekaka added the sig/planner SIG: Planner label May 13, 2021

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 13, 2021

address comment

818d532

qw4990 approved these changes May 13, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 13, 2021

Merge branch 'master' into stats-leak

cf7b6d4

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 13, 2021

ti-chi-bot added 4 commits May 13, 2021 14:13

Merge branch 'master' into stats-leak

981e3c5

Merge branch 'master' into stats-leak

abdeccd

Merge branch 'master' into stats-leak

96b7f23

Merge branch 'master' into stats-leak

36f6992

Merge branch 'master' into stats-leak

e3f24f1

ti-chi-bot added 2 commits May 13, 2021 17:49

Merge branch 'master' into stats-leak

4bea4de

Merge branch 'master' into stats-leak

ca76788

ti-chi-bot merged commit 1c4fbfc into pingcap:master May 13, 2021

tiancaiamao deleted the stats-leak branch May 14, 2021 01:56

tiancaiamao added needs-cherry-pick-release-5.0 needs-cherry-pick-4.0 labels May 14, 2021

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request May 14, 2021

cherry pick pingcap#24339 to release-5.0

a2747d3

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request May 14, 2021

server: close the temporary session in HTTP API to avoid memory leak (#24339) #24649

Merged

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request May 14, 2021

cherry pick pingcap#24339 to release-4.0

92d6d77

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request May 14, 2021

server: close the temporary session in HTTP API to avoid memory leak (#24339) #24650

Merged

ti-chi-bot pushed a commit that referenced this pull request Jul 8, 2021

server: close the temporary session in HTTP API to avoid memory leak (#…

2016134

…24339) (#24650)

ti-chi-bot pushed a commit that referenced this pull request Jul 8, 2021

server: close the temporary session in HTTP API to avoid memory leak (#…

7aefdd3

…24339) (#24649)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server: close the temporary session in HTTP API to avoid memory leak #24339

server: close the temporary session in HTTP API to avoid memory leak #24339

tiancaiamao commented Apr 28, 2021

eurekaka Apr 28, 2021 •

edited

tiancaiamao Apr 28, 2021

qw4990 Apr 29, 2021 •

edited

qw4990 left a comment •

edited

tiancaiamao commented Apr 29, 2021

tiancaiamao commented May 12, 2021

eurekaka commented May 13, 2021

ti-chi-bot commented May 13, 2021

eurekaka commented May 13, 2021

ti-chi-bot commented May 13, 2021

qw4990 commented May 13, 2021

ti-chi-bot commented May 13, 2021

ti-chi-bot commented May 13, 2021

ti-chi-bot commented May 13, 2021

ti-srebot commented May 14, 2021

ti-srebot commented May 14, 2021

server: close the temporary session in HTTP API to avoid memory leak #24339

server: close the temporary session in HTTP API to avoid memory leak #24339

Conversation

tiancaiamao commented Apr 28, 2021

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

eurekaka Apr 28, 2021 • edited

Choose a reason for hiding this comment

tiancaiamao Apr 28, 2021

Choose a reason for hiding this comment

qw4990 Apr 29, 2021 • edited

Choose a reason for hiding this comment

qw4990 left a comment • edited

Choose a reason for hiding this comment

tiancaiamao commented Apr 29, 2021

tiancaiamao commented May 12, 2021

eurekaka commented May 13, 2021

ti-chi-bot commented May 13, 2021

eurekaka commented May 13, 2021

ti-chi-bot commented May 13, 2021

qw4990 commented May 13, 2021

ti-chi-bot commented May 13, 2021

ti-chi-bot commented May 13, 2021

ti-chi-bot commented May 13, 2021

ti-srebot commented May 14, 2021

ti-srebot commented May 14, 2021

eurekaka Apr 28, 2021 •

edited

qw4990 Apr 29, 2021 •

edited

qw4990 left a comment •

edited