Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix panic in cassandra storage while registering metrics #2457

Merged
merged 1 commit into from
Jul 31, 2020

Conversation

adityacs
Copy link
Collaborator

What this PR does / why we need it:
Fixes panic in cassandra storage while registering metrics.

Cassandra storage panics on registering prometheus metrics as there is a mismatch in labels. Loki was missing component="table-manager-store in Desc{fqName: "gocql_connection_pool_hosts", help: "Current number of hosts in the connection pool.", constLabels: {client="table-manager"}, variableLabels: []}

panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "gocql_connection_pool_hosts", help: "Current number of hosts in the connection pool.", constLabels: {client="index-read",component="index-store-2020-06-11"}, variableLabels: []} has different label names or a different help string

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc00097d440, 0xc00096cc40, 0x1, 0x1)
	/pkg/mod/github.com/prometheus/client_golang@v1.7.1/prometheus/wrap.go:102 +0x195
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGaugeFunc(0x3b112a0, 0xc00097d440, 0x0, 0x0, 0x0, 0x0, 0x35151ca, 0x1b, 0x3555642, 0x2f, ...)
	/pkg/mod/github.com/prometheus/client_golang@v1.7.1/prometheus/promauto/auto.go:319 +0x13c
github.com/gocql/gocql.newPolicyConnPool(0x3ac0ba0, 0xc00097d3e0, 0x3b112a0, 0xc00097d440, 0xc0000b3400, 0xc0006901e0)
	/pkg/mod/github.com/grafana/gocql@v0.0.0-20200605141915-ba5dc39ece85/connectionpool.go:132 +0x22f
github.com/gocql/gocql.PoolConfig.buildPool(...)
	/pkg/mod/github.com/grafana/gocql@v0.0.0-20200605141915-ba5dc39ece85/cluster.go:28
github.com/gocql/gocql.NewSession(0xc00096cba0, 0x1, 0x1, 0x34de229, 0x5, 0x0, 0x77359400, 0x12a05f200, 0x2352, 0xc000ab46b8, ...)
	/pkg/mod/github.com/grafana/gocql@v0.0.0-20200605141915-ba5dc39ece85/session.go:136 +0x55f
github.com/gocql/gocql.(*ClusterConfig).CreateSession(...)
	/pkg/mod/github.com/grafana/gocql@v0.0.0-20200605141915-ba5dc39ece85/cluster.go:207
github.com/cortexproject/cortex/pkg/chunk/cassandra.(*Config).session(0xc000b762b8, 0x34e8b4f, 0xa, 0x3b112a0, 0xc00097d2f0, 0x0, 0x10bc238, 0xc000b764f8)
	/pkg/mod/github.com/cortexproject/cortex@v1.2.1-0.20200727121049-4cfa4a2978c2/pkg/chunk/cassandra/storage_client.go:125 +0x6bc
github.com/cortexproject/cortex/pkg/chunk/cassandra.NewStorageClient(0xc000ab46a0, 0xe, 0x2352, 0xc000ab46b8, 0x4, 0x34e0035, 0x6, 0x1, 0x10000, 0x0, ...)
	/pkg/mod/github.com/cortexproject/cortex@v1.2.1-0.20200727121049-4cfa4a2978c2/pkg/chunk/cassandra/storage_client.go:228 +0x88
github.com/cortexproject/cortex/pkg/chunk/storage.NewIndexClient(0xc000ab45f0, 0x9, 0x34e05ff, 0x6, 0x0, 0x4000000000000000, 0x4024000000000000, 0x0, 0x0, 0x186a0, ...)
	/pkg/mod/github.com/cortexproject/cortex@v1.2.1-0.20200727121049-4cfa4a2978c2/pkg/chunk/storage/factory.go:218 +0x9fa
github.com/cortexproject/cortex/pkg/chunk/storage.NewStore(0x34e05ff, 0x6, 0x0, 0x4000000000000000, 0x4024000000000000, 0x0, 0x0, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, ...)
	/pkg/mod/github.com/cortexproject/cortex@v1.2.1-0.20200727121049-4cfa4a2978c2/pkg/chunk/storage/factory.go:159 +0x675
github.com/grafana/loki/pkg/storage.NewStore(0x34e05ff, 0x6, 0x0, 0x4000000000000000, 0x4024000000000000, 0x0, 0x0, 0x186a0, 0x3ff4cccccccccccd, 0x3ff0000000000000, ...)
	/src/github.com/grafana/loki/pkg/storage/store.go:85 +0x116
github.com/grafana/loki/pkg/loki.(*Loki).initStore(0xc000179000, 0x3b5c8c0, 0xc0000c0e00, 0xc00097d020, 0xc00074e8f0)
	/src/github.com/grafana/loki/pkg/loki/modules.go:271 +0x33a
github.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices(0xc00023ee60, 0x34dcd10, 0x3, 0x3b554e0, 0xc0000e7680, 0xc0000e7680)
	/pkg/mod/github.com/cortexproject/cortex@v1.2.1-0.20200727121049-4cfa4a2978c2/pkg/util/modules/modules.go:87 +0x269
github.com/grafana/loki/pkg/loki.(*Loki).Run(0xc000179000, 0xc000b0c6c0, 0x4)
	/src/github.com/grafana/loki/pkg/loki/loki.go:178 +0x5b
main.main()
	/src/github.com/grafana/loki/cmd/loki/main.go:104 +0x61f

@codecov-commenter
Copy link

Codecov Report

Merging #2457 into master will decrease coverage by 0.05%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2457      +/-   ##
==========================================
- Coverage   62.79%   62.73%   -0.06%     
==========================================
  Files         162      162              
  Lines       13963    13964       +1     
==========================================
- Hits         8768     8761       -7     
- Misses       4513     4518       +5     
- Partials      682      685       +3     
Impacted Files Coverage Δ
pkg/loki/modules.go 5.02% <0.00%> (-0.03%) ⬇️
pkg/promtail/targets/file/tailer.go 73.86% <0.00%> (-2.28%) ⬇️
pkg/promtail/targets/file/filetarget.go 67.85% <0.00%> (-1.79%) ⬇️
pkg/logql/evaluator.go 92.47% <0.00%> (-0.41%) ⬇️

@adityacs adityacs merged commit 0c4f704 into grafana:master Jul 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants