distributor dies when enabling ha-tracker

[friedrich-at-adobe]

Steps

1. Start consul as daemon
2. Start one ingester and kill it
3. Start distributor with ha-tracker enabled

Results

• Distributors crashes

Expected

• No crashes from distributor

$ docker run -d -p 8500:8500 consul:1.6.0 agent  -ui -server -client=0.0.0.0 -bootstrap
$ go run cmd/cortex/main.go -target=ingester -dynamodb.url=inmemory:///cortex -chunk.storage-client=inmemory
level=info ts=2019-08-29T10:04:42.186978Z caller=cortex.go:198 msg=initialising module=overrides
level=info ts=2019-08-29T10:04:42.187492Z caller=override.go:53 msg="per-tenant overrides disabled"
level=info ts=2019-08-29T10:04:42.187525Z caller=cortex.go:198 msg=initialising module=store
level=info ts=2019-08-29T10:04:42.187601Z caller=cortex.go:198 msg=initialising module=server
level=info ts=2019-08-29T10:04:42.18787Z caller=server.go:117 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-08-29T10:04:42.188101Z caller=cortex.go:198 msg=initialising module=ingester
level=warn ts=2019-08-29T10:04:42.188262Z caller=net.go:15 msg="error getting interface" inf=eth0 err="route ip+net: no such network interface"
level=info ts=2019-08-29T10:04:42.188601Z caller=main.go:68 msg="Starting Cortex" version="(version=, branch=, revision=)"
level=info ts=2019-08-29T10:04:42.194753Z caller=lifecycler.go:387 msg="entry not found in ring, adding with no tokens"
level=info ts=2019-08-29T10:04:42.202317Z caller=lifecycler.go:317 msg="auto-joining cluster after timeout"
^Clevel=info ts=2019-08-29T10:05:25.553445Z caller=signals.go:54 msg="=== received SIGINT/SIGTERM ===\n*** exiting"
level=info ts=2019-08-29T10:05:25.553543Z caller=cortex.go:224 msg=stopping module=ingester
level=info ts=2019-08-29T10:05:25.553572Z caller=lifecycler.go:470 msg="changing ingester state from" old_state=ACTIVE new_state=LEAVING
level=info ts=2019-08-29T10:05:25.568159Z caller=transfer.go:212 msg="nothing to transfer"
level=info ts=2019-08-29T10:05:55.581966Z caller=lifecycler.go:368 msg="ingester removed from consul"
level=info ts=2019-08-29T10:05:55.582005Z caller=lifecycler.go:292 msg="member.loop() exited gracefully"
level=info ts=2019-08-29T10:05:55.582027Z caller=cortex.go:224 msg=stopping module=server
level=info ts=2019-08-29T10:05:55.582129Z caller=cortex.go:224 msg=stopping module=store
level=info ts=2019-08-29T10:05:55.582199Z caller=cortex.go:224 msg=stopping module=overrides
$ go run cmd/cortex/main.go -target=distributor -distributor.ha-tracker.enable=true
level=info ts=2019-08-29T10:06:06.348326Z caller=cortex.go:198 msg=initialising module=server
level=info ts=2019-08-29T10:06:06.348505Z caller=server.go:117 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-08-29T10:06:06.348959Z caller=cortex.go:198 msg=initialising module=overrides
level=info ts=2019-08-29T10:06:06.349088Z caller=override.go:53 msg="per-tenant overrides disabled"
level=info ts=2019-08-29T10:06:06.349171Z caller=cortex.go:198 msg=initialising module=ring
level=info ts=2019-08-29T10:06:06.349312Z caller=cortex.go:198 msg=initialising module=distributor
level=info ts=2019-08-29T10:06:06.349684Z caller=main.go:68 msg="Starting Cortex" version="(version=, branch=, revision=)"
panic: runtime error: index out of range

goroutine 386 [running]:
github.com/cortexproject/cortex/pkg/distributor.(*haTracker).loop.func1(0xc000106b6b, 0x4, 0x35fa940, 0xc0006de3a0, 0x0)
	/source/cortexproject/cortex/pkg/distributor/ha_tracker.go:157 +0x586
github.com/cortexproject/cortex/pkg/ring/kv.(*prefixedKVClient).WatchPrefix.func1(0xc000106b6b, 0x4, 0x35fa940, 0xc0006de3a0, 0x35fa940)
	/source/cortexproject/cortex/pkg/ring/kv/prefix.go:33 +0x85
github.com/cortexproject/cortex/pkg/ring/kv/consul.(*Client).WatchPrefix(0xc0005cc4b0, 0x3ba7c20, 0xc00064f860, 0xc0000551a0, 0xb, 0xc0002bcee0)
	/source/cortexproject/cortex/pkg/ring/kv/consul/client.go:246 +0x3f1
github.com/cortexproject/cortex/pkg/ring/kv.(*prefixedKVClient).WatchPrefix(0xc0006de020, 0x3ba7c20, 0xc00064f860, 0x0, 0x0, 0xc00044dbc0)
	/source/cortexproject/cortex/pkg/ring/kv/prefix.go:32 +0x166
github.com/cortexproject/cortex/pkg/ring/kv.metrics.WatchPrefix.func1(0x3ba7c20, 0xc00064f860, 0xb, 0xbf5206c394d8e618)
	/source/cortexproject/cortex/pkg/ring/kv/metrics.go:50 +0x5e
github.com/weaveworks/common/instrument.CollectedRequest(0x3ba7b60, 0xc0006ca000, 0x36e1d67, 0xb, 0x3b8a820, 0xc0000f6148, 0x37d1848, 0xc000094f00, 0x10, 0xc00044dbc0)
	/Volumes/cs/go/pkg/mod/github.com/weaveworks/common@v0.0.0-20190822150010-afb9996716e4/instrument/instrument.go:151 +0x26d
github.com/cortexproject/cortex/pkg/ring/kv.metrics.WatchPrefix(0x3ba81a0, 0xc0006de020, 0x3ba7b60, 0xc0006ca000, 0x0, 0x0, 0xc00044dbc0)
	/source/cortexproject/cortex/pkg/ring/kv/metrics.go:49 +0xef
github.com/cortexproject/cortex/pkg/distributor.(*haTracker).loop(0xc0006d4000, 0x3ba7b60, 0xc0006ca000)
	/source/cortexproject/cortex/pkg/distributor/ha_tracker.go:148 +0xbc
created by github.com/cortexproject/cortex/pkg/distributor.newClusterTracker
	/source/cortexproject/cortex/pkg/distributor/ha_tracker.go:138 +0x3bb
exit status 2

[weeco]

DId you check whether this is related to #1652 ?

[friedrich-at-adobe]

I confirm the error has been fixed in e16a10b. Thank you @gouthamve @weeco !