Prometheus 2.0: panic: runtime error: index out of range

[iobestar]

What did you do?
Nothing special.
What did you expect to see?
That Prometheus keep running.
What did you see instead? Under which circumstances?
Prometheus crashes.
Prometheus version:
prometheus, version 2.0.0-beta.1 (branch: HEAD, revision: 4dcb465)
build user: root@d3f9974fac5a
build date: 20170811-12:16:25
go version: go1.8.3

Logs:

panic: runtime error: index out of range

goroutine 1528 [running]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).AddFast(0xc6b1956a80, 0xc67fe61428, 0x8, 0x15e03a15cba, 0x3ff0000000000000, 0x8, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:436 +0x34e
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add(0xc6b1956a80, 0xc6b4d0ca20, 0x9, 0x9, 0x15e03a15cba, 0x3ff0000000000000, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:376 +0x57d
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add(0xc6b19256b0, 0xc6b4d0ca20, 0x9, 0x9, 0x15e03a15cba, 0x3ff0000000000000, 0xc6b4d1cbe0, 0x7, 0x6, 0xc6b4d01600)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:659 +0xbf
github.com/prometheus/prometheus/storage/tsdb.appender.Add(0x288a040, 0xc6b19256b0, 0xc6b4d0ca20, 0x9, 0x9, 0x15e03a15cba, 0x3ff0000000000000, 0xc6b4d1cbe0, 0x0, 0x9, ...)
	/go/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:133 +0xb4
github.com/prometheus/prometheus/storage/tsdb.(*appender).Add(0xc6b181b1b0, 0xc6b4d0ca20, 0x9, 0x9, 0x15e03a15cba, 0x3ff0000000000000, 0x5, 0x5, 0x5, 0x4)
	<autogenerated>:7 +0x97
github.com/prometheus/prometheus/storage.(*fanoutAppender).Add(0xc6b19256e0, 0xc6b4d0ca20, 0x9, 0x9, 0x15e03a15cba, 0x3ff0000000000000, 0xc6b4d0fc20, 0xc6b4d0ca20, 0x9, 0x9)
	/go/src/github.com/prometheus/prometheus/storage/fanout.go:105 +0x80
github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add(0xc6b19704a0, 0xc6b4d0ca20, 0x9, 0x9, 0x15e03a15cba, 0x3ff0000000000000, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:239 +0x7e
github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Add(0x2889900, 0xc6b19704a0, 0xc6b1968600, 0x5, 0x8, 0xc6b1c0b780, 0x4, 0x4, 0x15e03a15cba, 0x3ff0000000000000, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:274 +0x2bb
github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Add(0xc6b1925710, 0xc6b1c0b780, 0x4, 0x4, 0x15e03a15cba, 0x3ff0000000000000, 0x0, 0x0, 0x0, 0x0)
	<autogenerated>:52 +0xb6
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc511c72380, 0xc6b4a84000, 0x1efca2, 0x203e00, 0xed12c7e84, 0x10dd5db4, 0x294c1e0, 0x15e036ae7e1, 0x40ef440000000000, 0x0, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:830 +0x478
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc511c72380, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:646 +0x48c
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:302 +0x534

[onorua]

Would you mind updating to v2.0.0-beta.2 and check whether it is still reproducible?

[iobestar]

Prometheus is up 12 hours without exception. So far is ok.

[onorua]

Glad to hear, give it another 12 hours, if you feel the bug is not reproducible with the beta2 release, please close the bug ;)

[iobestar]

Same/Similar exception occurs again:

panic: runtime error: index out of range

goroutine 173272905 [running]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).AddFast(0xc73e975320, 0xc704af5098, 0x8, 0x15e7061fc62, 0x7ff0000000000002, 0x8, 0xe44b9ab89307d0df)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:436 +0x34e
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add(0xc73e975320, 0xc4215cfb80, 0xb, 0xb, 0x15e7061fc62, 0x7ff0000000000002, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:376 +0x57d
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add(0xc5e43e8390, 0xc4215cfb80, 0xb, 0xb, 0x15e7061fc62, 0x7ff0000000000002, 0xc4ee6e8260, 0xa, 0x9, 0xc852ec9300)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:659 +0xbf
github.com/prometheus/prometheus/storage/tsdb.appender.Add(0x288a040, 0xc5e43e8390, 0xc4215cfb80, 0xb, 0xb, 0x15e7061fc62, 0x7ff0000000000002, 0xc4ee6e8260, 0x0, 0xb, ...)
	/go/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:133 +0xb4
github.com/prometheus/prometheus/storage/tsdb.(*appender).Add(0xc723858170, 0xc4215cfb80, 0xb, 0xb, 0x15e7061fc62, 0x7ff0000000000002, 0x7, 0xa, 0x7, 0x4)
	<autogenerated>:7 +0x97
github.com/prometheus/prometheus/storage.(*fanoutAppender).Add(0xc5e43e83c0, 0xc4215cfb80, 0xb, 0xb, 0x15e7061fc62, 0x7ff0000000000002, 0xc66926a280, 0xc4215cfb80, 0xb, 0xb)
	/go/src/github.com/prometheus/prometheus/storage/fanout.go:105 +0x80
github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add(0xc4ee6e8200, 0xc4215cfb80, 0xb, 0xb, 0x15e7061fc62, 0x7ff0000000000002, 0x0, 0xc66926a140, 0x0, 0x18)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:239 +0x7e
github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Add(0x2889900, 0xc4ee6e8200, 0xc66926a140, 0x7, 0xa, 0xc59b23c000, 0x4, 0x4, 0x15e7061fc62, 0x7ff0000000000002, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/target.go:274 +0x2bb
github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Add(0xc5e43e83f0, 0xc59b23c000, 0x4, 0x4, 0x15e7061fc62, 0x7ff0000000000002, 0xc704af55b0, 0xc5e43e83f0, 0x0, 0x3047c284)
	<autogenerated>:52 +0xb6
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append.func1(0xc59b23c000, 0x4, 0x4, 0x296cfe0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:889 +0x8a
github.com/prometheus/prometheus/retrieval.(*scrapeCache).forEachStale(0xc74730f500, 0xc704af59a8)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:565 +0x126
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc702a8ca80, 0x296c0e8, 0x0, 0x0, 0xed14855b7, 0x1783b0f4, 0x294c1e0, 0x3, 0x3, 0x0, ...)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:899 +0xbb6
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).endOfRunStaleness(0xc702a8ca80, 0xed1485599, 0x1783c557, 0x294c1e0, 0xc4cbde1200, 0x6fc23ac00)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:718 +0x39c
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc702a8ca80, 0x6fc23ac00, 0x2540be400, 0x0)
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:675 +0xa0b
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func3
	/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:235 +0x82

[onorua]

@iobestar version is beta2?

[iobestar]

Yes, beta2 (sorry for not mentioning).

[WeiBanjo]

Getting same crash on beta 3.

$ prometheus --version
prometheus, version 2.0.0-beta.3 (branch: HEAD, revision: 066783b3991dd64729325fc4f880dfffb484a2c2)
  build user:       root@0cbc320660dc
  build date:       20170912-10:17:45
  go version:       go1.8.3

prometheus[6291]: panic: runtime error: invalid memory address or nil pointer dereference
prometheus[6291]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x167b22e]
prometheus[6291]: goroutine 204 [running]:
prometheus[6291]: github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc47bcec680, 0xc4f13b4000, 0x13c25ff, 0x1b898d1, 0xed14a4d75, 0x326cbc5a, 0x2965460, 0x0, 0x0, 0x0, ...)
prometheus[6291]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:801 +0x18fe
prometheus[6291]: github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc47bcec680, 0xdf8475800, 0xdf8475800, 0x0)
prometheus[6291]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:668 +0x560
prometheus[6291]: created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
prometheus[6291]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:305 +0x534

[fabxc]

Thanks for reporting. The original issue should no longer exist as the code path was removed.
This trace is pointing to different location.

[fabxc]

Closing this as a dupe of #2955

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.