(Another) panic: runtime error: invalid memory address or nil pointer dereference

[ichekrygin]

Reopen of #2339

What did you do?
Run Prometheus

What did you expect to see?
Prometheus is up and running

What did you see instead? Under which circumstances?
Prometheus crashed

Environment

• System information:

  Linux 4.7.3-coreos-r2 x86_64

• Prometheus version:

  prometheus, version 1.5.2 (branch: master, revision: bd1182d)
  build user: root@1a01c5f68840
  build date: 20170210-16:23:28
  go version: go1.7.5

• Alertmanager version:

  N/A

• Prometheus configuration file:

    global:
      scrape_interval: 30s
      scrape_timeout: 30s

    rule_files:
    - /etc/prometheus/alert.rules

    scrape_configs:
    - job_name: etcd
      static_configs:
        - targets:
          - 10.72.132.6:2379
          - 10.72.134.15:2379
          - 10.72.146.27:2379
          - 10.72.145.36:2379
          - 10.72.144.241:2379

    - job_name: 'prometheus'
      static_configs:
        - targets: ['localhost:9090']

    - job_name: 'kube-state-metrics'
      static_configs:
        - targets: ['prometheus-kube-state-metrics:8080']

    - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

    - job_name: 'kube-dns-dnsmasq'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: http
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: kube-system;kube-dns;metrics-sidecar

    - job_name: 'kube-dns-skydns'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: http
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: kube-system;kube-dns;metrics-kubedns

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    - job_name: 'kubernetes-service-endpoints'
      scheme: https
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    - job_name: 'kubernetes-services'
      scheme: https
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name

• Alertmanager configuration file:

N/A

• Logs:

time="2017-03-07T18:09:38Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:612" 
time="2017-03-07T18:10:01Z" level=info msg="Done checkpointing in-memory metrics and chunks in 22.980738545s." source="persistence.go:639" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe pc=0x57ad22]

goroutine 640 [running]:
panic(0x18b2120, 0xc420016030)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/prometheus/prometheus/storage/local/chunk.(*Desc).Add(0x6, 0x15aa9fa1597, 0x4147580000000000, 0x0, 0x0, 0x0, 0x5020105, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/local/chunk/chunk.go:136 +0x22
github.com/prometheus/prometheus/storage/local.(*memorySeries).add(0xc4ae5f3490, 0x15aa9fa1597, 0x4147580000000000, 0xc4ae5f3490, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/local/series.go:238 +0x115
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Append(0xc42038a6e0, 0xc5b48ee4c0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/local/storage.go:877 +0x398
github.com/prometheus/prometheus/storage.Fanout.Append(0xc420536740, 0x2, 0x2, 0xc5b48ee4c0, 0xc4aff35488, 0x19e3620)
        /go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0x66
github.com/prometheus/prometheus/storage.(*Fanout).Append(0xc4205369c0, 0xc5b48ee4c0, 0xc4b1b23440, 0x27257d0)
        <autogenerated>:3 +0x6e
github.com/prometheus/prometheus/retrieval.(*countingAppender).Append(0xc51b7396c0, 0xc5b48ee4c0, 0xc4a750fa28, 0xc4a750fa18)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:301 +0x48
github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Append(0x26876c0, 0xc51b7396c0, 0xc4d96b0000, 0xc5b48ee4c0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:241 +0x1b3
github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Append(0xc51b739700, 0xc5b48ee4c0, 0x0, 0x0)
        <autogenerated>:35 +0x6e
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc4808cc230, 0xc4cfd16000, 0x7616, 0x9400, 0x1aaa6c5b, 0x27309c0, 0xc4cfd16000)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:503 +0x2f1
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc4808cc230, 0x6fc23ac00, 0x6fc23ac00, 0x0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:413 +0x5c3
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:247 +0x3a4

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.