web: runtime error: invalid memory address or nil pointer dereference

[Dominik-K]

What did you do?
We didn't use the Prometheus website directly. Grafana queries Prometheus constantly for our live dashboard.

What did you expect to see?
Normal resource usage, no panic log ;-)

What did you see instead? Under which circumstances?

Prometheus logged this panic:

level=error ts=2017-12-05T09:22:36.84401431Z caller=stdlib.go:89 component=web caller="http: panic serving 10.0.0.226:55730" msg="runtime error: invalid memory address or nil pointer dereference"

, but seems to recover it. Around 11 minutes later, it logged many query timeouts:

level=warn ts=2017-12-05T09:33:34.155483363Z caller=manager.go:312 component="rule manager" […] msg="Evaluating rule failed" […] err="query timed out in query queue"

Directly after the panic, the memory usage raised and the CPU usage dropped significantly:

Prometheus was running since 2017-12-01 19:11 UTC (86h11m).

The same error happened today:

level=error ts=2017-12-08T10:37:30.750758496Z caller=stdlib.go:89 component=web caller="http: panic serving 10.0.0.226:49188" msg="runtime error: invalid memory address or nil pointer dereference"
level=warn ts=2017-12-08T10:49:30.384532964Z caller=manager.go:312 component="rule manager" […] msg="Evaluating rule failed" […] err="query timed out in query queue"

Prometheus was running since 2017-12-07 12:54:27 UTC (19h43m).

Environment

• System information:

  Inside container: Linux 4.4.15-25.57.amzn1.x86_64 x86_64

• Prometheus version:

  Docker image: prom/prometheus v2.0.0 Image ID: 67141fa03496 created 4 weeks ago

  prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98)
  build user: root@615b82cb36b6
  build date: 20171108-07:11:59
  go version: go1.9.2

• Prometheus configuration file:
  Will provide them if necessary.

• Logs:
  see above

It's maybe related to #3412

[brian-brazil]

This smells like a null pointer down in storage.

[simonpasquier]

This issue being reported against v2.0.0, can you reproduce it with the latest Prometheus version (v2.3.2)?

[krasi-georgiev]

since it is such an old issue and haven't received a reply lets close and will revisit if needed.

[Dominik-K]

It didn't occur again, currently running on v2.3.0. It wasn't reproducible, either. So, it's fine that you closed it 👍

[krasi-georgiev]

thanks, appreciated the update!