crash: Remote storage resharding from 55 to 7 shards, panic: runtime error: makeslice: len out of range

[iksaif]

Looks like the remote storage code can make prometheus crash somtime:

time="2017-09-21T15:55:36Z" level=warning msg="Error sending 100 samples to remote storage: context deadline exceeded" source="queue_manager.go:500" 
time="2017-09-21T15:55:36Z" level=warning msg="Error sending 100 samples to remote storage: context deadline exceeded" source="queue_manager.go:500" 
time="2017-09-21T15:55:36Z" level=warning msg="Error sending 100 samples to remote storage: context deadline exceeded" source="queue_manager.go:500" 
time="2017-09-21T15:55:36Z" level=warning msg="Error sending 100 samples to remote storage: context deadline exceeded" source="queue_manager.go:500" 
time="2017-09-21T15:55:36Z" level=warning msg="Error sending 100 samples to remote storage: context deadline exceeded" source="queue_manager.go:500" 
time="2017-09-21T15:55:44Z" level=info msg="Remote storage resharding from 55 to 7 shards." source="queue_manager.go:351" 
panic: runtime error: makeslice: len out of range

goroutine 239 [running]:
github.com/prometheus/prometheus/storage/remote.(*QueueManager).newShards(0xc4202e1960, 0xfffffffffffffff7, 0x1)
	/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:396 +0x40
github.com/prometheus/prometheus/storage/remote.(*QueueManager).reshard(0xc4202e1960, 0xfffffffffffffff7)
	/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:375 +0xcf
github.com/prometheus/prometheus/storage/remote.(*QueueManager).reshardLoop(0xc4202e1960)
	/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:364 +0x105
created by github.com/prometheus/prometheus/storage/remote.(*QueueManager).Start
	/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:265 +0x85
2017/09/21 15:55:54.433860 [DEBUG] (cli) receiving signal "child exited"
2017/09/21 15:55:54.434032 [INFO] (runner) child process died
Consul Template returned errors:
child process died with exit code 2

[tomwilkie]

What version of Prometheus are you running? I suspect this might be fixed in 1.7.2

[iksaif]

This was 1.7.1 I believe. Will try to deploy 1.7.2 and re-open if it happens again.

[tomwilkie]

Thanks Corentin!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.