horrible high cpu #614
Description
Describe the bug
It has been used for a while, but suddenly at a certain point in time, after the exporter reported an error indicating that it could not find relevant data, it retried crazily, causing the exporter to occupy an abnormally high cpu, nearly 100. The specific error is as follows logs
To Reproduce
can't reproduce
Expected behavior
Although I don't know what caused the connection to fail, it shouldn't take up such a high cpu. If you can't query the data, you can query it later. It shouldn't be so frequent
Logs
Jan 10 19:28:33 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:33+08:00" level=error msg="Failed to get $dbstats for database ttc_gjfs: connection(localhost:27018[-236893]) incomplete read of full message: context canceled"
Jan 10 19:28:33 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:33+08:00" level=error msg="Failed to get $dbstats for database ttc_payroll: timed out while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
Jan 10 19:28:33 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:33+08:00" level=error msg="Failed to get $dbstats for database ttc_xxjg: timed out while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0"
Jan 10 19:28:43 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:43+08:00" level=error msg="cannot get replSetGetStatus: connection(localhost:27018[-237165]) incomplete read of full message: context canceled"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="cannot get replSetGetStatus: connection(localhost:27018[-237332]) incomplete read of full message: context deadline exceeded"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="Failed to get $dbstats for database ttc_gjfs: connection(localhost:27018[-237350]) incomplete read of full message: context deadline exceeded"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="Failed to get $dbstats for database ttc_xxjg: connection(localhost:27018[-237344]) incomplete read of full message: context deadline exceeded"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="Failed to get $dbstats for database ttc_payroll: connection(localhost:27018[-237343]) incomplete read of full message: context deadline exceeded"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="Failed to get $dbstats for database ttc_payroll: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="Failed to get $dbstats for database ttc_xxjg: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="cannot get replSetGetStatus: context deadline exceeded"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: time="2023-01-10T19:28:50+08:00" level=error msg="Failed to get $dbstats for database ttc_xxjg: timed out while checking out a connection from connection pool: context deadline exceeded; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 1"
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: 2023/01/10 19:28:50 http: panic serving 10.10.22.18:60232: descriptor Desc{fqName: "", help: "", constLabels: {}, variableLabels: []} is invalid: connection(localhost:27018[-237339]) incomplete read of full message: context deadline exceeded
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: goroutine 11906415 [running]:
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: net/http.(*conn).serve.func1()
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /opt/hostedtoolcache/go/1.17.8/x64/src/net/http/server.go:1802 +0xb9
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: panic({0xae9700, 0xc004ac3e90})
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /opt/hostedtoolcache/go/1.17.8/x64/src/runtime/panic.go:1047 +0x266
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xd501e0, {0xc00070b880, 0x1, 0x8})
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /home/runner/go/pkg/mod/github.com/prometheus/client_golang@v1.12.1/prometheus/registry.go:403 +0x7f
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: github.com/percona/mongodb_exporter/exporter.(*Exporter).makeRegistry(0xc000075ae0, {0xd501e0, 0xc001633da0}, 0xc003625c70, {0xd4b908, 0xc00a7ad440}, {{0xc00016c8e0, 0x1, 0x1}, 0x0, ...})
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/exporter.go:193 +0xbc5
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: github.com/percona/mongodb_exporter/exporter.(*Exporter).Handler.func1({0xd4d5b8, 0xc001def5e0}, 0xc002fe8e00)
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /home/runner/work/mongodb_exporter/mongodb_exporter/exporter/exporter.go:304 +0x405
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: net/http.HandlerFunc.ServeHTTP(0x0, {0xd4d5b8, 0xc001def5e0}, 0x463e4e)
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /opt/hostedtoolcache/go/1.17.8/x64/src/net/http/server.go:2047 +0x2f
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: net/http.serverHandler.ServeHTTP({0xc006e14ff0}, {0xd4d5b8, 0xc001def5e0}, 0xc002fe8e00)
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /opt/hostedtoolcache/go/1.17.8/x64/src/net/http/server.go:2879 +0x43b
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: net/http.(*conn).serve(0xc00052b900, {0xd50218, 0xc000296120})
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: /opt/hostedtoolcache/go/1.17.8/x64/src/net/http/server.go:1930 +0xb08
Jan 10 19:28:50 LinkFinMongodb02 mongodb_exporter[107796]: created by net/http.(*Server).Serve
This is only a part, and most of the subsequent logs will repeat this part
Environment
- OS: Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) Upstream updates bugfixes #1 SMP Tue Nov 22 16:42:41 UTC 2016
- environment:k8s
- MongoDB version:3.6.3
- mongodb_exporter version:0.32.0
Additional context
