http connection leak #30571

tiancaiamao · 2021-12-09T06:38:00Z

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

In the master branch we have the lint check for http close #28755

In 5.1 5.2 it's not added...
and some user meet the connection leak issues like https://asktug.com/t/topic/183219/16

2. What did you expect to see? (Required)

golangci-lint run -c ./.golangci.yml --disable-all -E bodyclose

3. What did you see instead (Required)

Some error detected, for example in 5.1.1

4. What is your TiDB version? (Required)

The text was updated successfully, but these errors were encountered:

tiancaiamao · 2021-12-09T06:39:02Z

A hotfix for 5.1.1 is #30570

We need to check the 5.1 5.2 5.3 branch and fix all of them

tiancaiamao · 2021-12-09T10:05:41Z

https://github.com/pingcap/tidb/pull/30570/files#diff-d19169611c4613e5d3f09dc49da1c5d99bc6c7adb90a4b2ebbd75ac869806f50R697

This is the most important change, it visits "/pd/api/v1/config"

One possible case call stack could be from telemetry, and the rate is 1min/op

goroutine 643 [running]:
runtime/debug.Stack(0x0, 0x26, 0x0)
        /home/genius/project/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /home/genius/project/go/src/runtime/debug/stack.go:16 +0x25
github.com/pingcap/tidb/domain/infosync.(*InfoSyncer).getPrometheusAddr(0xc0005a6880, 0x13c33cad3a0, 0x55b5ce0, 0x551e70799, 0x7f4c0c2d9e68)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:700 +0x1b0
github.com/pingcap/tidb/domain/infosync.GetPrometheusAddr(0x203000, 0x8, 0x55eeb48, 0x203000)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:661 +0xb2
github.com/pingcap/tidb/telemetry.querySQLMetric(0x3c14568, 0xc000054090, 0xc06492c52bbc21b2, 0x14185b1d689, 0x55b5ce0, 0x37af363, 0x2e, 0x0, 0x0, 0x0, ...)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:123 +0xa5
github.com/pingcap/tidb/telemetry.readSQLMetric(0xc06492c52bbc21b2, 0x14185b1d689, 0x55b5ce0, 0xc00113a068, 0x2, 0xc000dbdf8c)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:109 +0x70
github.com/pingcap/tidb/telemetry.RotateSubWindow()
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:191 +0x293
github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc00014e870)
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1100 +0x65
created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1089 +0x7f

Other possible way to trigger it is visit the METRICS_SCHEMA tables, for example,

use METRICS_SCHEMA;
select * from up;

The call stack is

goroutine 643 [running]:
runtime/debug.Stack(0x0, 0x26, 0x0)
        /home/genius/project/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /home/genius/project/go/src/runtime/debug/stack.go:16 +0x25
github.com/pingcap/tidb/domain/infosync.(*InfoSyncer).getPrometheusAddr(0xc0005a6880, 0x1796773891a, 0x55b5ce0, 0xdf85e98bc, 0xc0018ff680)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:700 +0x1b0
github.com/pingcap/tidb/domain/infosync.GetPrometheusAddr(0x203000, 0x8, 0x0, 0x203000)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:661 +0xb2
github.com/pingcap/tidb/telemetry.querySQLMetric(0x3c14568, 0xc000054090, 0xc06493102c77b059, 0x1875fd21d30, 0x55b5ce0, 0x37af363, 0x2e, 0x0, 0x0, 0x0, ...)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:123 +0xa5
github.com/pingcap/tidb/telemetry.readSQLMetric(0xc06493102c77b059, 0x1875fd21d30, 0x55b5ce0, 0xc000653f68, 0x2, 0xc0009a4f8c)
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:109 +0x70
github.com/pingcap/tidb/telemetry.RotateSubWindow()
        /home/genius/project/src/github.com/pingcap/tidb/telemetry/data_window.go:191 +0x293
github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc00014e870)
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1100 +0x65
created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop
        /home/genius/project/src/github.com/pingcap/tidb/domain/domain.go:1089 +0x7f
goroutine 12831 [running]:
runtime/debug.Stack(0x0, 0x26, 0x0)
        /home/genius/project/go/src/runtime/debug/stack.go:24 +0x9f
runtime/debug.PrintStack()
        /home/genius/project/go/src/runtime/debug/stack.go:16 +0x25
github.com/pingcap/tidb/domain/infosync.(*InfoSyncer).getPrometheusAddr(0xc0005a6880, 0x1875fe2bf7d, 0x55b5ce0, 0x2df12fdcb, 0x38)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:700 +0x1b0
github.com/pingcap/tidb/domain/infosync.GetPrometheusAddr(0x2dee1ec6c1ea2561, 0x2c91b58, 0x2d0000c000092c00, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/domain/infosync/info.go:661 +0xb2
github.com/pingcap/tidb/executor.(*MetricRetriever).queryMetric(0xc001a52d60, 0x3c145d8, 0xc001130300, 0x3c53fd0, 0xc00134c000, 0xc064927d44bc2a21, 0xfe8c2968f8, 0x55b5ce0, 0xc064931344bc2a21, 0x18a3ef2d8f8, ...)
        /home/genius/project/src/github.com/pingcap/tidb/executor/metrics_reader.go:99 +0xa5
github.com/pingcap/tidb/executor.(*MetricRetriever).retrieve(0xc001a52d60, 0x3c145d8, 0xc001130300, 0x3c53fd0, 0xc00134c000, 0x0, 0x0, 0xc000a44c28, 0x116d6dd, 0xc00076cfc0)
        /home/genius/project/src/github.com/pingcap/tidb/executor/metrics_reader.go:77 +0x2ef
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0xc000879710, 0x3c145d8, 0xc001130300, 0xc00189b5e0, 0xc0003f7c70, 0xc00118a000)
        /home/genius/project/src/github.com/pingcap/tidb/executor/memtable_reader.go:113 +0x497
github.com/pingcap/tidb/executor.Next(0x3c145d8, 0xc001130300, 0x3c18838, 0xc000879710, 0xc00189b5e0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2de
github.com/pingcap/tidb/executor.(*recordSet).Next(0xc00189b540, 0x3c145d8, 0xc001130300, 0xc00189b5e0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/executor/adapter.go:145 +0xb5
github.com/pingcap/tidb/server.(*tidbResultSet).Next(0xc00189b590, 0x3c145d8, 0xc001130300, 0xc00189b5e0, 0xc00137e360, 0x400)
        /home/genius/project/src/github.com/pingcap/tidb/server/driver_tidb.go:304 +0x51
github.com/pingcap/tidb/server.(*clientConn).writeChunks(0xc000de0100, 0x3c145d8, 0xc001130300, 0x3c2ac60, 0xc00189b590, 0x20000, 0x55e8de0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1875 +0x4de
github.com/pingcap/tidb/server.(*clientConn).writeResultset(0xc000de0100, 0x3c145d8, 0xc001130300, 0x3c2ac60, 0xc00189b590, 0x2ac00, 0x0, 0x0, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1823 +0x20e
github.com/pingcap/tidb/server.(*clientConn).handleStmt(0xc000de0100, 0x3c14530, 0xc001130300, 0x3c2eca8, 0xc001976780, 0x55e8de0, 0x0, 0x0, 0x1, 0x0, ...)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1717 +0x3e5
github.com/pingcap/tidb/server.(*clientConn).handleQuery(0xc000de0100, 0x3c14530, 0xc00076cfc0, 0xc0010a2379, 0x10, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1563 +0x492
github.com/pingcap/tidb/server.(*clientConn).dispatch(0xc000de0100, 0x3c14530, 0xc00076cfc0, 0xc0010a2378, 0x11, 0x10, 0x0, 0x0)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:1097 +0xafd
github.com/pingcap/tidb/server.(*clientConn).Run(0xc000de0100, 0x3c145d8, 0xc001a28180)
        /home/genius/project/src/github.com/pingcap/tidb/server/conn.go:860 +0x296
github.com/pingcap/tidb/server.(*Server).onConn(0xc000f7e340, 0xc000de0100)
        /home/genius/project/src/github.com/pingcap/tidb/server/server.go:485 +0xa7e
created by github.com/pingcap/tidb/server.(*Server).Run

bb7133 · 2021-12-10T02:30:31Z

@tiancaiamao Can we close this issue because of #30462 and #28756? Usually when an issue is fixed in master branch, we can close it and treat it as backlog for backporting for release branch.

tiancaiamao · 2021-12-10T02:37:56Z

OK, now I've verified the following actions:

Hotfix for 5.1.1
Make sure we have the lint check on master branch so we can prevent this kind of error in the future
Check and cherry pick to 5.1
Check and cherry pick to 5.2
Check and cherry pick to 5.3, just a tiny fix, 5.3 is not really affected

tiancaiamao · 2021-12-10T02:43:33Z

@tiancaiamao Can we close this issue because of #30462 and #28756? Usually when an issue is fixed in master branch, we can close it and treat it as backlog for backporting for release branch.

I think we can close this one after the cherry-picks are merged, or if you're worried about the increase of DI, maybe we can close it now and make the following fixes asynchronous. @bb7133

github-actions · 2021-12-10T02:44:49Z

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.

close #30571

tiancaiamao added type/bug The issue is confirmed as a bug. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. affects-5.3 This bug affects 5.3.x versions. labels Dec 9, 2021

tiancaiamao self-assigned this Dec 9, 2021

tiancaiamao added the severity/major label Dec 9, 2021

ChenPeng2013 added the sig/sql-infra SIG: SQL Infra label Dec 9, 2021

tiancaiamao mentioned this issue Dec 9, 2021

*: fix some http connection leaks found by lint check #30570

Merged

12 tasks

This was referenced Dec 10, 2021

*: fix some http connection leaks found by lint check #30597

Merged

*: fix some http connection leaks found by lint check #30599

Merged

tiancaiamao mentioned this issue Dec 10, 2021

*: fix some http connection leaks found by lint check #30600

Merged

12 tasks

tiancaiamao closed this as completed Dec 10, 2021

github-actions bot added the needs-more-info label Dec 10, 2021

shuke987 added severity/critical and removed severity/major labels Dec 14, 2021

ti-chi-bot pushed a commit that referenced this issue Apr 15, 2022

*: fix some http connection leaks found by lint check (#30599)

fbf9144

close #30571

ti-chi-bot pushed a commit that referenced this issue Jun 16, 2022

*: fix some http connection leaks found by lint check (#30600)

de2d3f5

close #30571

XuHuaiyu mentioned this issue Jun 22, 2022

releases: add v5.3.2 release notes pingcap/docs-cn#9914

Merged

13 tasks

ti-chi-bot pushed a commit that referenced this issue Aug 12, 2022

*: fix some http connection leaks found by lint check (#30597)

1406e26

close #30571

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http connection leak #30571

http connection leak #30571

tiancaiamao commented Dec 9, 2021

tiancaiamao commented Dec 9, 2021

tiancaiamao commented Dec 9, 2021

bb7133 commented Dec 10, 2021

tiancaiamao commented Dec 10, 2021

tiancaiamao commented Dec 10, 2021

github-actions bot commented Dec 10, 2021

http connection leak #30571

http connection leak #30571

Comments

tiancaiamao commented Dec 9, 2021

Bug Report

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

tiancaiamao commented Dec 9, 2021

tiancaiamao commented Dec 9, 2021

bb7133 commented Dec 10, 2021

tiancaiamao commented Dec 10, 2021

tiancaiamao commented Dec 10, 2021

github-actions bot commented Dec 10, 2021