Crash on startup with duplicate metrics collector registration

[bboreham]

As of commit 59bbc46, in ingester, querier and ruler.
Beginning of log:

level=info ts=2020-03-19T17:56:09.556287259Z caller=server.go:147 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0000ce550, 0xc000838670, 0x1, 0x1)
	/go/pkg/mod/github.com/prometheus/client_golang@v1.5.0/prometheus/registry.go:400 +0xad
github.com/thanos-io/thanos/pkg/discovery/dns.NewProvider(0x3a17800, 0xc00013b650, 0x3a70400, 0xc0000ce550, 0x3388a7a, 0x6, 0xc0008385f0)
	/go/pkg/mod/github.com/thanos-io/thanos@v0.11.0/pkg/discovery/dns/provider.go:74 +0x36a
github.com/cortexproject/cortex/pkg/chunk/cache.NewMemcachedClient(0x7ffd811bb6a1, 0x26, 0x338f0c5, 0x9, 0x0, 0x0, 0x5f5e100, 0x40, 0xdf8475800, 0x0, ...)
	/go/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached_client.go:101 +0x19c

I guess we need integration tests that include memcached.

[gouthamve]

Yes, this wasn't caught before because none of the integration tests add 2 memcache instances. I've added the fix and tests in #2303

[bboreham]

Still crashing, albeit slightly different place:

level=info ts=2020-03-20T13:50:46.704894927Z caller=server.go:147 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2020-03-20T13:50:46.706289307Z caller=schema_config.go:89 msg="running with DEPRECATED flag -config-yaml, use -schema-config-file instead"
panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc000565f50, 0xc0004eb4a0, 0x1, 0x1)
	/go/pkg/mod/github.com/prometheus/client_golang@v1.5.0/prometheus/wrap.go:93 +0x170
github.com/thanos-io/thanos/pkg/discovery/dns.NewProvider(0x3a18860, 0xc0009e4210, 0x3a714a0, 0xc000565f50, 0x3389a7a, 0x6, 0xc0004eb400)
	/go/pkg/mod/github.com/thanos-io/thanos@v0.11.0/pkg/discovery/dns/provider.go:74 +0x36a
github.com/cortexproject/cortex/pkg/chunk/cache.NewMemcachedClient(0x7fffbc70a43e, 0x2a, 0x7fffbc70a492, 0x9, 0x0, 0x0, 0x3b9aca00, 0x20, 0xdf8475800, 0x0, ...)
	/go/src/github.com/cortexproject/cortex/pkg/chunk/cache/memcached_client.go:105 +0x2b1
github.com/cortexproject/cortex/pkg/chunk/cache.New(0x0, 0x0, 0xa, 0x2710, 0x0, 0x400, 0x10, 0x7fffbc70a43e, 0x2a, 0x7fffbc70a492, ...)
	/go/src/github.com/cortexproject/cortex/pkg/chunk/cache/cache.go:84 +0x296
github.com/cortexproject/cortex/pkg/chunk.NewChunkFetcher(0x0, 0x0, 0xa, 0x2710, 0x0, 0x400, 0x10, 0x7fffbc70a43e, 0x2a, 0x7fffbc70a492, ...)
	/go/src/github.com/cortexproject/cortex/pkg/chunk/chunk_store_utils.go:104 +0x70
github.com/cortexproject/cortex/pkg/chunk.newStore(0x0, 0x0, 0xa, 0x2710, 0x0, 0x400, 0x10, 0x7fffbc70a43e, 0x2a, 0x7fffbc70a492, ...)
	/go/src/github.com/cortexproject/cortex/pkg/chunk/chunk_store.go:88 +0x84
github.com/cortexproject/cortex/pkg/chunk.(*CompositeStore).AddPeriod(0xc0006f1318, 0x0, 0x0, 0xa, 0x2710, 0x0, 0x400, 0x10, 0x7fffbc70a43e, 0x2a, ...)
	/go/src/github.com/cortexproject/cortex/pkg/chunk/composite_store.go:67 +0x370
github.com/cortexproject/cortex/pkg/chunk/storage.NewStore(0x3389774, 0x6, 0xc00041e880, 0x4000000000000000, 0x4024000000000000, 0x0, 0x0, 0x0, 0x186a0, 0x3ff4cccccccccccd, ...)
...

querier run as:

        image: cortexproject/cortex:master-931b89e5
        args:
        - -target=querier
        - -server.http-listen-port=80
        - -consul.hostname=consul.cortex.svc.cluster.local.:8500
        - -config-yaml=/etc/config/schema.yaml
        - -querier.frontend-address=query-frontend.cortex.svc.cluster.local.:9095
        - -s3.url=s3://redacted
        - -dynamodb.url=dynamodb://redacted
        - -querier.query-store-after=10m
        - -memcached.hostname=memcached-chunks.cortex.svc.cluster.local.
        - -memcached.timeout=1s
        - -memcached.service=memcached
        - -memcached.batchsize=1024
        - -memcached.parallelism=16
        - -memcached.max-idle-conns=32 
        - -store.index-cache-read.cache.enable-fifocache
        - -store.index-cache-read.fifocache.size=10240
        - -store.index-cache-read.fifocache.duration=10m
        - -store.index-cache-validity=10m
        - -store.index-cache-read.memcached.hostname=memcached-qf.cortex.svc.cluster.local.
        - -store.index-cache-read.memcached.expiration=10m
        - -store.index-cache-read.memcached.timeout=1s
        - -store.index-cache-read.memcached.batchsize=1024
        - -store.index-cache-read.memcached.parallelism=16
        - -store.index-cache-read.memcached.max-idle-conns=32 
        - -store.max-query-length=169h 
        - -querier.worker-parallelism=1
        - -querier.batch-iterators=true
        - -querier.ingester-streaming=true
        - -querier.query-ingesters-within=16h
        - -distributor.extra-query-delay=25ms
        - -mem-ballast-size-bytes=268435456