index out of range on scrape

[auhlig]

What did you do?

Prometheus in Kubernetes v1.4.6 federating metrics crashed with below error. So far not reproducible.

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

• System information:

  Linux 4.14.11-coreos x86_64

• Prometheus version:

  prometheus, version 2.0.0 (branch: master, revision: e4df16e1a965628987eab78315a50ed21535f73b)
  build user: root@5756d655d3ce
  build date: 20171204-15:26:24
  go version: go1.9.2

• Prometheus configuration file:

https://github.com/sapcc/helm-charts/blob/master/system/kube-monitoring/charts/prometheus-frontend/templates/config.yaml#L13-L124

• Logs:

level=info ts=2018-01-11T12:17:41.915163872Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.0.0, branch=master, revision=e4df16e1a965628987eab78315a50ed21535f73b)"
level=info ts=2018-01-11T12:17:41.915335124Z caller=main.go:221 build_context="(go=go1.9.2, user=root@5756d655d3ce, date=20171204-15:26:24)"
level=info ts=2018-01-11T12:17:41.915391347Z caller=main.go:222 host_details="(Linux 4.14.11-coreos #1 SMP Fri Jan 5 11:00:14 UTC 2018 x86_64 prometheus-frontend-3790183880-lubx3 (none))"
level=info ts=2018-01-11T12:17:41.918789913Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2018-01-11T12:17:41.918931203Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-01-11T12:17:41.918809679Z caller=main.go:399 msg="Starting TSDB ..."
level=info ts=2018-01-11T12:17:58.043037287Z caller=main.go:409 msg="TSDB started"
level=info ts=2018-01-11T12:17:58.043373994Z caller=main.go:491 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yaml
level=info ts=2018-01-11T12:17:58.067890664Z caller=main.go:386 msg="Server is ready to receive requests."
panic: runtime error: index out of range

goroutine 616 [running]:
github.com/prometheus/prometheus/pkg/textparse.(*Parser).Metric(0xc471c39cc8, 0xc471c39c80, 0xc47e132967, 0xbe)
/go/src/github.com/prometheus/prometheus/pkg/textparse/parse.go:125 +0x77c
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc437eb8f80, 0xc47a200000, 0x400d968, 0x545fe00, 0xbe8df364c1f07062, 0xa87272a4c, 0x297a220, 0x0, 0x15ef3e5, 0x0, ...)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:842 +0x4ab
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc437eb8f80, 0x6fc23ac00, 0x5d21dba00, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:671 +0x691
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:289 +0x2fc

[gouthamve]

Could have been closed by #3793

[auhlig]

Indeed. Never saw the issue again.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.