invalid memory address or nil pointer dereference

[errordeveloper]

What did you do?

I ran the latest build from master, and it paniced.

What did you expect to see?

Happy Prometheus.

What did you see instead? Under which circumstances?

level=info ts=2018-01-04T12:34:30.797481431Z caller=main.go:221 msg="Starting Prometheus" version="(version=2.0.0, branch=master, revision=ec94df49d45baa8e680d81e732294eb04fff00de)"
level=info ts=2018-01-04T12:34:30.797587605Z caller=main.go:222 build_context="(go=go1.9.2, user=root@1885792ecf46, date=20171229-20:03:34)"
level=info ts=2018-01-04T12:34:30.797618333Z caller=main.go:223 host_details="(Linux 4.4.64+ #1 SMP Wed Aug 30 20:27:36 PDT 2017 x86_64 weave-cortex-agent-1352299515-q9b40 (none))"
level=info ts=2018-01-04T12:34:30.797643323Z caller=main.go:224 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2018-01-04T12:34:30.802483848Z caller=manager.go:58 component="scrape manager" msg="Starting scrape manager..."
level=info ts=2018-01-04T12:34:30.802561868Z caller=main.go:434 msg="Starting TSDB ..."
level=info ts=2018-01-04T12:34:30.806440473Z caller=web.go:383 component=web msg="Start listening for connections" address=:8080
level=info ts=2018-01-04T12:34:30.812878794Z caller=main.go:444 msg="TSDB started"
level=info ts=2018-01-04T12:34:30.812978586Z caller=main.go:511 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-01-04T12:34:30.814840891Z caller=kubernetes.go:191 component="discovery manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-01-04T12:34:30.816859216Z caller=kubernetes.go:191 component="discovery manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-01-04T12:34:30.817553956Z caller=kubernetes.go:191 component="discovery manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-01-04T12:34:30.818090353Z caller=kubernetes.go:191 component="discovery manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-01-04T12:34:30.81978125Z caller=kubernetes.go:191 component="discovery manager" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2018-01-04T12:34:30.821172572Z caller=main.go:421 msg="Server is ready to receive requests."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x155d32a]

goroutine 123 [running]:
github.com/prometheus/prometheus/discovery.(*Manager).allGroups.func1(0x29062c0, 0xc420402c00)
	/go/src/github.com/prometheus/prometheus/discovery/manager.go:188 +0x28a
github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc4203a9700, 0x29062c0, 0xc420402c00, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/manager.go:96 +0x54
main.main.func5(0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:338 +0x52
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc420052240, 0xc42047fb60, 0xc4204424c0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
	/go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

Environment

Kubernetes 1.7 on GKE.

• Prometheus version:

> docker run prom/prometheus@sha256:6050e90c156d4e6be31a23e87b53673b2a2d2ab5b71677225555675b43a1b445 --version                 
prometheus, version 2.0.0 (branch: master, revision: ec94df49d45baa8e680d81e732294eb04fff00de)
  build user:       root@1885792ecf46
  build date:       20171229-20:03:34
  go version:       go1.9.2

• Prometheus configuration file:

        global:
          scrape_interval: 5s
        remote_write:
          - url: 'https://cloud.weave.works./api/prom/push'
            basic_auth:
              password: <snip>
        remote_read:
          - url: 'https://cloud.weave.works./api/prom/read'
            basic_auth:
              password: <snip>
        scrape_configs:
          - job_name: kubernetes-apiservers
            kubernetes_sd_configs:
              - role: endpoints
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
              - source_labels:
                  - __meta_kubernetes_namespace
                  - __meta_kubernetes_service_name
                  - __meta_kubernetes_endpoint_port_name
                action: keep
                regex: default;kubernetes;https
              - source_labels:
                  - __meta_kubernetes_namespace
                target_label: _weave_namespace
              - source_labels:
                  - __meta_kubernetes_endpoints_name
                target_label: _weave_service
          - job_name: kubernetes-pods
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels:
                  - __meta_kubernetes_pod_annotation_prometheus_io_scrape
                action: drop
                regex: 'false'
              - source_labels:
                  - __meta_kubernetes_pod_annotation_prometheus_io_scheme
                action: replace
                target_label: __scheme__
                regex: ^(https?)$
                replacement: $1
              - source_labels:
                  - __meta_kubernetes_pod_annotation_prometheus_io_path
                action: replace
                target_label: __metrics_path__
                regex: ^(.+)$
                replacement: $1
              - source_labels:
                  - __address__
                  - __meta_kubernetes_pod_annotation_prometheus_io_port
                action: replace
                target_label: __address__
                regex: '([^:]+)(?::\d+)?;(\d+)'
                replacement: '$1:$2'
              - source_labels:
                  - __meta_kubernetes_namespace
                target_label: _weave_namespace
              - source_labels:
                  - __meta_kubernetes_pod_name
                  - __meta_kubernetes_pod_node_name
                target_label: node
                regex: ^weave-cortex-node-exporter-.+;(.+)$
                replacement: $1
              - source_labels:
                  - _weave_service
                  - __meta_kubernetes_pod_name
                target_label: _weave_service
                regex: '^;(kube-.*)-(?:ip|gke)-.*$'
                replacement: $1
              - source_labels:
                  - _weave_service
                  - __meta_kubernetes_pod_name
                target_label: _weave_service
                regex: '^;(.*?)(?:(?:-[0-9bcdf]+)?-[0-9a-z]{5})?$'
                replacement: $1
              - source_labels:
                  - _weave_service
                  - __meta_kubernetes_pod_name
                regex: ^;(.+)$
                target_label: _weave_service
                replacement: $1
          - job_name: kubernetes-nodes
            kubernetes_sd_configs:
              - role: node
            tls_config:
              insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
              - target_label: __scheme__
                replacement: https
              - target_label: __address__
                replacement: 'kubernetes.default.svc:443'
              - source_labels:
                  - __meta_kubernetes_node_name
                regex: (.+)
                target_label: __metrics_path__
                replacement: '/api/v1/nodes/${1}/proxy/metrics'
              - target_label: _weave_namespace
                replacement: default
              - target_label: _weave_service
                replacement: kubelet
          - job_name: weave-net
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels:
                  - __meta_kubernetes_namespace
                  - __meta_kubernetes_pod_label_name
                action: keep
                regex: ^kube-system;weave-net$
              - source_labels:
                  - __meta_kubernetes_pod_container_name
                  - __address__
                action: replace
                target_label: __address__
                regex: '^weave;(.+?)(?::\d+)?$'
                replacement: '$1:6782'
              - source_labels:
                  - __meta_kubernetes_pod_container_name
                  - __address__
                action: replace
                target_label: __address__
                regex: '^weave-npc;(.+?)(?::\d+)?$'
                replacement: '$1:6781'
              - source_labels:
                  - __meta_kubernetes_namespace
                target_label: _weave_namespace
              - target_label: _weave_service
                replacement: weave-net
          - job_name: cadvisor
            kubernetes_sd_configs:
              - role: node
            tls_config:
              insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            scheme: https
            relabel_configs:
              - target_label: __address__
                replacement: 'kubernetes.default.svc:443'
              - source_labels:
                  - __meta_kubernetes_node_name
                regex: (.+)
                target_label: __metrics_path__
                replacement: '/api/v1/nodes/${1}/proxy/metrics/cadvisor'
              - target_label: _weave_namespace
                replacement: default
              - target_label: _weave_service
                replacement: cadvisor
            metric_relabel_configs:
              - source_labels:
                  - _weave_pod_name
                  - pod_name
                target_label: _weave_pod_name
                regex: '^;(kube-.*)-(?:ip|gke)-.*$'
                replacement: $1
              - source_labels:
                  - _weave_pod_name
                  - pod_name
                target_label: _weave_pod_name
                regex: '^;(.*?)(?:(?:-[0-9bcdf]+)?-[0-9a-z]{5})?$'
                replacement: $1
              - source_labels:
                  - _weave_pod_name
                  - pod_name
                regex: ^;(.+)$
                target_label: _weave_pod_name
                replacement: $1

[errordeveloper]

It doesn't panic when I use prom/prometheus@sha256:b6e5078fdd6ba9abb61455448859217b715c9f63d745c2ac8874c506c6cfb98d (which corresponds to #3592).

[brian-brazil]

Sounds like this is already fixed then and will be in 2.1.0, but it would be by some other PR.

[errordeveloper]

@brian-brazil I think you've misread what I'm saying here. Master is broken.

[brian-brazil]

Ah. @krasi-georgiev this sounds like what you're working on at the moment.

[krasi-georgiev]

@errordeveloper at first glance this looks like a different bug in the role: endpoints, but the k8s discovery was broken after I did the general service discovery refactoring.
Anyway the PR above should fix both issues if you want to test it and let me know.

[stanxing]

I have the same problem

[krasi-georgiev]

@stanxing can you try the mentioned PR to see if it fixes it for you ?

[chalky9909]

@brian-brazil I see you have referenced an updated release 2.1.0 in the comments above.

We have encountered the same panic (below) from the latest master release which we are currently troubleshooting:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x155d32a]

We are using the :master release tagged quay.io/prometheus/prometheus SHA256: 0b745814b3c5

Do you have an approximate ETA for when the 2.10 release may be available that may resolve this issue?

[krasi-georgiev]

@chalky9909 hard to tell if it is the same issue without the full stack trace , but if it really is than I am pretty sure the PR reference above fixes this issue so once it is merged should be good to go.
If you could test it to confirm that would be great.

Otherwise if your full stack trace looks different please open a new issue.

[chalky9909]

@krasi-georgiev I have tested the PR reference above (Revision : a8cce41882e5d94f876d93c1b7c29d37a32f58eb) and this has not reported the same panic so does look to have resolved the issue.

[bamb00]

Hi @krasi-georgiev,

What is the best way to reproduce this error with prometheus server?

Thanks.

[krasi-georgiev]

@bamb00 first time someone wants to know a best way to reproduce a bug 😄

you should be able to reproduce the bug by using the config form the first comment and build the ec94df4 Prometheus revision.