mimir crashed with target=all when frontend memcached enabled.

[masonmei]

Describe the bug

When running mimir with target=all, and the frontend memcached enabled. The stack as below:

level=info ts=2022-04-14T07:37:51.939711217Z caller=main.go:193 msg="Starting application" version="(version=2.0.0, branch=HEAD, revision=9fd2da5)"
level=info ts=2022-04-14T07:37:51.941383397Z caller=server.go:285 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-04-14T07:37:51.943109066Z caller=memberlist_client.go:394 msg="Using memberlist cluster node name" name=mimir-instance-2-084e7ca7
level=info ts=2022-04-14T07:37:51.946416605Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-04-14T07:37:51.948110455Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-04-14T07:37:51.951153947Z caller=memcached.go:60 msg="created memcached index cache"
level=info ts=2022-04-14T07:37:51.952584352Z caller=memberlist_client.go:513 msg="joined memberlist cluster" reached_nodes=3
level=info ts=2022-04-14T07:37:51.954534494Z caller=memcached.go:49 msg="created memcached cache"
level=info ts=2022-04-14T07:37:51.956163091Z caller=memcached.go:49 msg="created memcached cache"
panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "thanos_memcached_dns_provider_results", help: "The number of resolved endpoints for each configured address", constLabels: {name="frontend-cache"}, variableLabels: [addr]} has different label names or a different help string

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc000c66ff0, {0xc0006dc3f0, 0x1, 0x40d3e7})
/__w/mimir/mimir/vendor/github.com/prometheus/client_golang/prometheus/wrap.go:104 +0x151
github.com/thanos-io/thanos/pkg/extprom.NewTxGaugeVec({0x20fc680, 0xc000c66ff0}, {{0x0, 0x0}, {0x0, 0x0}, {0x1c57cbb, 0x14}, {0x1cc1bd2, 0x3c}, ...}, ...)
/__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/extprom/tx_gauge.go:42 +0x212
github.com/thanos-io/thanos/pkg/discovery/dns.NewProvider({0x20dec60, 0xc00001b630}, {0x20fc680, 0xc000c66ff0}, {0x1c4109b, 0x7ff6ed77ddd8})
/__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/discovery/dns/provider.go:64 +0x187
github.com/thanos-io/thanos/pkg/cacheutil.newMemcachedClient({0x20dec60, 0xc00001b630}, {0x20f4c00, 0xc000a3f640}, 0xc000c66f30, {{0xc0006dc330, 0x1, 0x1}, 0xbebc200, 0x64, ...}, ...)
/__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/cacheutil/memcached_client.go:256 +0x1ca
github.com/thanos-io/thanos/pkg/cacheutil.NewMemcachedClientWithConfig({0x20dec60, 0xc00001b630}, {0x1c4b648, 0xe}, {{0xc0006dc330, 0x1, 0x1}, 0xbebc200, 0x64, 0x32, ...}, ...)
/__w/mimir/mimir/vendor/github.com/thanos-io/thanos/pkg/cacheutil/memcached_client.go:235 +0x295
github.com/grafana/mimir/pkg/cache.CreateClient({0x1c4b648, 0xe}, {{0xc00048a2a0, 0x9}, {{0xc0000529c0, 0x3f}, 0xbebc200, 0x64, 0x32, 0x61a8, ...}}, ...)
/__w/mimir/mimir/pkg/cache/cache.go:52 +0x188
github.com/grafana/mimir/pkg/frontend/querymiddleware.newResultsCache({{{0xc00048a2a0, 0x9}, {{0xc0000529c0, 0x3f}, 0xbebc200, 0x64, 0x32, 0x61a8, 0x64, 0x64, ...}}, ...}, ...)
/__w/mimir/mimir/pkg/frontend/querymiddleware/results_cache.go:82 +0x92
github.com/grafana/mimir/pkg/frontend/querymiddleware.newQueryTripperware({0x4e94914f0000, 0x1, {{{0xc00048a2a0, 0x9}, {{...}, 0xbebc200, 0x64, 0x32, 0x61a8, 0x64, ...}}, ...}, ...}, ...)
/__w/mimir/mimir/pkg/frontend/querymiddleware/roundtrip.go:177 +0x365
github.com/grafana/mimir/pkg/frontend/querymiddleware.NewTripperware({0x4e94914f0000, 0x1, {{{0xc00048a2a0, 0x9}, {{...}, 0xbebc200, 0x64, 0x32, 0x61a8, 0x64, ...}}, ...}, ...}, ...)
/__w/mimir/mimir/pkg/frontend/querymiddleware/roundtrip.go:138 +0xfe
github.com/grafana/mimir/pkg/mimir.(*Mimir).initQueryFrontendTripperware(0xc000590000)
/__w/mimir/mimir/pkg/mimir/modules.go:481 +0x1e5
github.com/grafana/dskit/modules.(*Manager).initModule(0xc000117830, {0x7ffddbcab46f, 0x1c72b3a}, 0x1e, 0x0)
/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:120 +0x224
github.com/grafana/dskit/modules.(*Manager).InitModuleServices(0x1984080, {0xc000344380, 0x1, 0x3})
/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:92 +0x10c
github.com/grafana/mimir/pkg/mimir.(*Mimir).Run(0xc000590000)
/__w/mimir/mimir/pkg/mimir/mimir.go:426 +0x21c
main.main()
/__w/mimir/mimir/cmd/mimir/main.go:195 +0xcd3

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (latest)

Run this with docker-compose mimir.tar.gz

Expected behavior

A clear and concise description of what you expected to happen.

Environment

• Infrastructure: [Kubernetes, docker-for-mac]
• Deployment tool: [helm, docker-compose]

Additional Context

[pracucci]

Thanks for your report! I'm looking into it.