Prometheus Release 2.2.0 Memory Leak ? #4164

piaoyu · May 15, 2018

Proposal

Use case. Why is this important?

Nice to have' is not a good use case :)

Bug Report

What did you do?
Not thing

What did you expect to see?
using memory and releasing it

What did you see instead? Under which circumstances?
using memory and not releasing it

Environment

System information:

#uname -srm
Linux 3.10.0-327.ali2010.rc7.alios7.x86_64 x86_64

Prometheus version:

/prometheus $ prometheus --version
prometheus, version 2.2.1 (branch: HEAD, revision: 94e4a4321761f83207ad11542ee971e7d5220e80)
  build user:       root@XXX-XXX-aXXX
  build date:       20180508-12:56:09
  go version:       go1.9.4

Alertmanager version:

/alertmanager $ alertmanager --version
alertmanager, version 0.14.0 (branch: HEAD, revision: 30af4d051b37ce817ea7e35b56c57a0e2ec9dbb0)
  build user:       root@37b6a49ebba9
  build date:       20180213-08:16:42
  go version:       go1.9.2

Prometheus configuration file:

# Source: prometheus/templates/server-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-0
  namespace: monitoring
  labels:
    component: "server"
    addonmanager.kubernetes.io/mode: Reconcile
data:
  prometheus.yml: |
    global:
      scrape_interval: 30s
      #scrape_timeout: 10s
      external_labels:
        zone: xxx

    #remote_write:
     #- url: http://remote-storage-adapter-service.monitoring.xxxx.local:9201/write

    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - prometheus-alertmanager-service.monitoring.xxxx.local:9093

    rule_files:
      - "/etc/rules/*.yml"
      - "/etc/rules/alerts"

    scrape_configs:

      - job_name: 'prometheus-server'
        static_configs:
        - targets:
          - xxxx:9090
          - xxxx:9090

      - job_name: 'kubernetes-apiservers'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
        - targets:
          - xxxx:6443
          - xxxx:6443
          - xxxx:6443
        metric_relabel_configs:
          - source_labels: ['__name__']
            regex: 'apiserver_request.*'
            action: keep

      - job_name: 'kubernetes-etcds'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
        - targets:
          - xxxx:2379
          - xxxx:2379
          - xxxx:2379

      - job_name: 'kubernetes-controller-manager'
        scheme: http
        static_configs:
        - targets:
          - xxxx:10252
          - xxxx:10252
          - xxxx:10252

      - job_name: 'kubernetes-kube-scheduler'
        scheme: http
        static_configs:
        - targets:
          - xxxx:10251
          - xxxx:10251
          - xxxx:10251

      - job_name: 'kubernetes-state-metrics'
        scheme: http
        static_configs:
        - targets:
          - prometheus-kube-state-metrics.monitoring.xxxx.local:8085
        metric_relabel_configs:
          - action: labeldrop
            regex: instance
          - action: labeldrop
            regex: container_id
          - action: labeldrop
            regex: image_id
          - source_labels: ['__name__']
            action: drop
            regex: '(kube_persistentvolume_status_phase|kube_pod_status_phase)'

      - job_name: 'kubernetes-nodes'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: xxxx.local:6443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}:4194/proxy/metrics
        metric_relabel_configs:
          - source_labels: ['__name__']
            action: keep
            regex: '(container_cpu_usage_seconds_total|container_memory.*|container_network_.*|container_fs.*|machine.*)'
          - source_labels: ['__name__']
            action: drop
            regex: container_memory_failures_total
          - source_labels: [pod_name]
            replacement: "$1"
            target_label: pod
          - action: labeldrop
            regex: id
          - action: labeldrop
            regex: name
          - action: labeldrop
            regex: beta_kubernetes_io_arch
          - action: labeldrop
            regex: beta_kubernetes_io_os

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: xxxx.local:6443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}:10255/proxy/metrics
        metric_relabel_configs:
          - source_labels: ['__name__']
            regex: kubelet_volume_stats_.*
            action: keep


      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: endpoints
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: (.+)(?::\d+);(\d+)
            replacement: ${1}:${2}
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name
        metric_relabel_configs:
          - source_labels: ['__name__']
            regex: node_hwmon_.*
            action: drop
          - source_labels: ['__name__']
            regex: http_.*
            action: drop
          - source_labels: ['__name__']
            regex: node_vmstat_.*
            action: drop

      - job_name: 'kubernetes-services'
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: service
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: kubernetes_name

      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - api_server: xxxx.local:6443
          role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: (.+):(?:\d+);(\d+)
            replacement: ${1}:${2}
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name
  rules: ""

Alertmanager configuration file:

insert configuration here (if relevant to the issue)

Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

piaoyu · May 15, 2018

(pprof) top 20
Showing nodes accounting for 8.79GB, 93.68% of 9.38GB total
Dropped 318 nodes (cum <= 0.05GB)
Showing top 20 nodes out of 86
      flat  flat%   sum%        cum   cum%
    2.45GB 26.14% 26.14%     2.52GB 26.85%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go
    0.76GB  8.10% 34.23%     0.76GB  8.10%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*decbuf).uvarintStr /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/encoding_helpers.go
    0.69GB  7.40% 41.63%     0.69GB  7.40%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.newReader.func2 /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go
    0.69GB  7.38% 49.00%     2.74GB 29.23%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go
    0.57GB  6.06% 55.06%     0.85GB  9.02%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceNodeCondition /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.50GB  5.32% 60.39%     3.24GB 34.55%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.47GB  5.02% 65.40%     0.74GB  7.85%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*Reader).readSymbols /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go
    0.43GB  4.61% 70.02%     0.45GB  4.84%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decResourceList /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.37GB  3.92% 73.94%     0.37GB  3.92%  github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels /root/gopath/src/github.com/prometheus/prometheus/pkg/labels/labels.go
    0.30GB  3.24% 77.18%     0.30GB  3.24%  github.com/prometheus/prometheus/pkg/textparse.(*Parser).Metric /root/gopath/src/github.com/prometheus/prometheus/pkg/textparse/parse.go
    0.23GB  2.46% 79.64%     0.23GB  2.46%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*decbuf).uvarintStr /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/encoding_helpers.go
    0.23GB  2.44% 82.08%     0.46GB  4.90%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*walReader).decodeSeries /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go
    0.21GB  2.29% 84.37%     0.21GB  2.29%  reflect.unsafe_New /root/working/go/src/runtime/malloc.go
    0.18GB  1.92% 86.29%     0.18GB  1.92%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc.NewXORChunk /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc/xor.go
    0.18GB  1.92% 88.21%     0.26GB  2.73%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecMapStringStringV /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go
    0.15GB  1.61% 89.82%     0.15GB  1.61%  github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness /root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go
    0.11GB  1.22% 91.05%     0.11GB  1.22%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.newMemSeries /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go
    0.10GB  1.02% 92.07%     0.10GB  1.02%  github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef /root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go
    0.08GB   0.9% 92.97%     0.08GB   0.9%  github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get /root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go
    0.07GB  0.71% 93.68%     0.07GB  0.71%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).string /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go (inline)

piaoyu · May 15, 2018

piaoyu · May 15, 2018

goroutine-stacks.txt

brancz · May 15, 2018

I think I’ve seen a memory profile like that before, can you just to try remove the scrape configs with “role: node” and see if it disappears? (I’ve seen leaks somehow related to node objects being deserialized, never found the reason for it though)

piaoyu · May 15, 2018

@brancz

just remove the role: node? (--config.file=/etc/config/prometheus.yml): parsing YAML file /etc/config/prometheus.yml: role missing (one of: pod, service, endpoints, node) .

I remove the kubernetes-service-endpoints  target which  has  about 150 node-exporter  and it's fine.
Is it the to many retrieval.(*scrapePool)  to cause OOM ?

brancz · May 15, 2018

Sorry I meant the whole block of those scrape configs that contain the “role: node” key/value pair.

krasi-georgiev · May 15, 2018

@cofyc do you have any idea what might be causing this leak?

cofyc · May 16, 2018

@krasi-georgiev I've never encountered memory leak with prometheus 2.2.x.
@piaoyu Could you increase your prometheus memory limit to see if it grows unboundedly? For current prometheus, each kubernetes discovery job creates its own client and informers (also cache store), so prometheus discovery may need a lot of memory. #2191 tracks this issue.

piaoyu · May 17, 2018

@cofyc Looks like it's memory also grows always .

Showing top 20 nodes out of 63
      flat  flat%   sum%        cum   cum%
14207.86MB 44.51% 44.51% 14647.38MB 45.89%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go
 3982.24MB 12.48% 56.99% 15910.02MB 49.85%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go
 3054.68MB  9.57% 66.56%  4629.74MB 14.50%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceNodeCondition /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
 2656.08MB  8.32% 74.88% 18566.11MB 58.17%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
 2424.38MB  7.60% 82.48%  2554.89MB  8.00%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decResourceList /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
 1214.31MB  3.80% 86.28%  1214.31MB  3.80%  reflect.unsafe_New /root/working/go/src/runtime/malloc.go
 1189.78MB  3.73% 90.01%  1678.80MB  5.26%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecMapStringStringV /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go
  467.66MB  1.47% 91.47%   467.66MB  1.47%  github.com/prometheus/prometheus/pkg/textparse.(*Parser).Metric /root/gopath/src/github.com/prometheus/prometheus/pkg/textparse/parse.go
  439.52MB  1.38% 92.85%   439.52MB  1.38%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).string /root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go (inline)
  429.69MB  1.35% 94.20%   429.69MB  1.35%  github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels /root/gopath/src/github.com/prometheus/prometheus/pkg/labels/labels.go
  362.51MB  1.14% 95.33%  1575.05MB  4.93%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeCondition).codecDecodeSelfFromMap /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
  111.51MB  0.35% 95.68%   225.51MB  0.71%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceNodeAddress /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
  101.01MB  0.32% 96.00%   162.01MB  0.51%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceTaint /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
      34MB  0.11% 96.10% 26627.76MB 83.42%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
      26MB 0.081% 96.19%   622.03MB  1.95%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeSystemInfo).codecDecodeSelfFromMap /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
   15.50MB 0.049% 96.23%   403.15MB  1.26%  github.com/prometheus/prometheus/pkg/relabel.relabel /root/gopath/src/github.com/prometheus/prometheus/pkg/relabel/relabel.go
       6MB 0.019% 96.25%   198.01MB  0.62%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeSpec).codecDecodeSelfFromMap /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
    0.50MB 0.0016% 96.25% 15910.02MB 49.85%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap /root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go
         0     0% 96.25%   403.15MB  1.26%  github.com/prometheus/prometheus/pkg/relabel.Process /root/gopath/src/github.com/prometheus/prometheus/pkg/relabel/relabel.go
         0     0% 96.25%  1455.62MB  4.56%  github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append /root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go

piaoyu · May 17, 2018

goroutine.txt

heap is to large for upload

piaoyu · May 17, 2018

heap profile: 63366: 323055784 [60591067: 597811480608] @ heap/1048576
61: 80953344 [61: 80953344] @ 0x40a135 0x40a2f3 0x408dee 0x158a3d7 0x1585301 0x158386f 0x1583508 0x158d866 0x159ab11 0x1599ab9 0x1598fba 0x45d051
#	0x158a3d6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.newReader.func2+0x146			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:634
#	0x1585300	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*Reader).readOffsetTable+0x210	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:798
#	0x158386e	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.newReader+0x31e			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:630
#	0x1583507	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.NewFileReader+0xe7			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:586
#	0x158d865	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.OpenBlock+0x155				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/block.go:256
#	0x159ab10	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0x380				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:486
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

61: 46972928 [61: 46972928] @ 0x40a135 0x40a2f3 0x40c562 0x1584fa2 0x15837dc 0x1583508 0x158d866 0x159ab11 0x1599ab9 0x1598fba 0x45d051
#	0x1584fa1	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*Reader).readSymbols+0x171	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:768
#	0x15837db	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.newReader+0x28b		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:615
#	0x1583507	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.NewFileReader+0xe7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/index.go:586
#	0x158d865	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.OpenBlock+0x155			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/block.go:256
#	0x159ab10	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0x380			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:486
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

2: 19251200 [18193: 175118540800] @ 0x15f34e4 0x15fb424 0x45d051
#	0x15f34e3	github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get+0xd3		/root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go:59
#	0x15fb423	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x503	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:661

1: 16547840 [1: 16547840] @ 0x11e69e7 0x11e61b0 0x11e5b66 0x11eb8de 0x11dec68 0x11dc540 0x11eb7ea 0xcb387e 0xcb3641 0xcb358d 0x11dc39e 0x11e4a30 0x45d051
#	0x11e69e6	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*processorListener).add+0x106										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:487
#	0x11e61af	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).distribute+0x10f									/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:392
#	0x11e5b65	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas+0x3d5									/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:350
#	0x11eb8dd	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas-fm+0x3d								/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:198
#	0x11dec67	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x267											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451
#	0x11dc53f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:147
#	0x11eb7e9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29	/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121
#	0xcb387d	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96
#	0xcb3640	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xa0											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97
#	0xcb358c	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c												/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52
#	0x11dc39d	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x22d											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121
#	0x11e4a2f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x37f										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:213

1: 15237120 [1: 15237120] @ 0x11e69e7 0x11e61b0 0x11e5b66 0x11eb8de 0x11dec68 0x11dc540 0x11eb7ea 0xcb387e 0xcb3641 0xcb358d 0x11dc39e 0x11e4a30 0x45d051
#	0x11e69e6	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*processorListener).add+0x106										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:487
#	0x11e61af	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).distribute+0x10f									/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:392
#	0x11e5b65	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas+0x3d5									/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:350
#	0x11eb8dd	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas-fm+0x3d								/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:198
#	0x11dec67	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x267											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451
#	0x11dc53f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:147
#	0x11eb7e9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29	/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121
#	0xcb387d	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96
#	0xcb3640	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xa0											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97
#	0xcb358c	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c												/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52
#	0x11dc39d	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x22d											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121
#	0x11e4a2f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x37f										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:213

6039: 10821888 [8887: 15925504] @ 0x1132d3c 0x10cf561 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0x1132d3b	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceNodeCondition+0x9ab			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71241
#	0x10cf560	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x7e0			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47518
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

34: 10305536 [81: 24551424] @ 0x40a135 0x40a2f3 0x40c962 0x15fe8e8 0x15fb5b1 0x45d051
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x1da7	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1da7		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:823
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

29: 8790016 [102: 30916608] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

4179: 8558592 [6136: 12566528] @ 0x11346fd 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0x11346fc	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x9bc		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71479
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

1: 8388608 [66: 553648128] @ 0x15b3183 0x15b4469 0x15b247f 0x15a2fbc 0x1599795 0x15be2ee 0x15c1591 0x15c2195 0x156b7f9 0x1604a3c 0x15fd647 0x15fb5b1 0x45d051
#	0x15b3182	bufio.NewWriterSize+0x1a2										/root/working/go/src/bufio/bufio.go:541
#	0x15b3182	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).cut+0x1a2		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:622
#	0x15b4468	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).write+0x48		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:759
#	0x15b247e	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).LogSamples+0xfe	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:473
#	0x15a2fbb	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Commit+0xeb		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:530
#	0x1599794	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.dbAppender.Commit+0x34		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:318
#	0x15be2ed	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Commit+0x4d		<autogenerated>:1
#	0x15c1590	github.com/prometheus/prometheus/storage/tsdb.appender.Commit+0x30					/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:258
#	0x15c2194	github.com/prometheus/prometheus/storage/tsdb.(*appender).Commit+0x44					<autogenerated>:1
#	0x156b7f8	github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit+0x48					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:161
#	0x1604a3b	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Commit+0x3b				<autogenerated>:1
#	0x15fd646	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0xb06					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:940
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

30: 6881280 [101: 23166976] @ 0x40a135 0x40a2f3 0x40d23d 0x15fe78d 0x15fb5b1 0x45d051
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addDropped+0x1c4c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:567
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1c4c		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:864
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 6422528 [31730: 101893406720] @ 0x15f34e4 0x15fb424 0x45d051
#	0x15f34e3	github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get+0xd3		/root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go:59
#	0x15fb423	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x503	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:661

22: 5046272 [80: 18350080] @ 0x40a135 0x40a2f3 0x40d23d 0x15fd18d 0x15fb5b1 0x45d051
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:562
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:903
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 4734976 [3: 14204928] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 4734976 [2: 9469952] @ 0x40a135 0x40a2f3 0x40c962 0x15fe8e8 0x15fb5b1 0x45d051
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x1da7	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1da7		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:823
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 3620864 [3: 10862592] @ 0x40a135 0x40a2f3 0x40d23d 0x15fd18d 0x15fb5b1 0x45d051
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:562
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:903
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

22: 3424256 [77: 11984896] @ 0x40a135 0x40a2f3 0x40c962 0x15fe8e8 0x15fb5b1 0x45d051
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x1da7	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1da7		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:823
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 3342336 [2: 6684672] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

7: 3211264 [17: 7798784] @ 0x40a135 0x40a2f3 0x40d23d 0x15fe78d 0x15fb5b1 0x45d051
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addDropped+0x1c4c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:567
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1c4c		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:864
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 3080192 [12: 18481152] @ 0x11277e4 0x10ab9bf 0x10ab6e5 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0x11d6bb7 0x11ea8ee 0x11e021d 0x11e1cb7 0x11ead33 0xcb387e 0xcb3641 0xcb358d 0x45d051
#	0x11277e3	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSlicePod+0x9c3				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:69575
#	0x10ab9be	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*PodList).codecDecodeSelfFromMap+0x27e			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:37478
#	0x10ab6e4	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*PodList).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:37395
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0x11d6bb6	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest.Result.Get+0x96							/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/request.go:1119
#	0x11ea8ed	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.NewListWatchFromClient.func1+0x1ed				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/listwatch.go:67
#	0x11e021c	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List+0x5c					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/listwatch.go:90
#	0x11e1cb6	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0x256				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/reflector.go:245
#	0x11ead32	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*Reflector).RunUntil.func1+0x32				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/reflector.go:200
#	0xcb387d	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96
#	0xcb3640	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xa0					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97
#	0xcb358c	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52

5: 2990080 [15: 8970240] @ 0x40a135 0x40a2f3 0x40c962 0x15fe8e8 0x15fb5b1 0x45d051
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x1da7	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1da7		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:823
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

5: 2990080 [20: 11960320] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

7: 2523136 [26891: 9692807168] @ 0x15f34e4 0x15fb424 0x45d051
#	0x15f34e3	github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get+0xd3		/root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go:59
#	0x15fb423	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x503	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:661

15: 2334720 [113: 17588224] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

20: 2293760 [82: 9404416] @ 0x40a135 0x40a2f3 0x40d23d 0x15fd18d 0x15fb5b1 0x45d051
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:562
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:903
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 2023424 [1: 2023424] @ 0x40a135 0x40a2f3 0x408dee 0x1588c74 0x1588aaf 0x15a54b7 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x1588c73	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).addFor+0x133	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:202
#	0x1588aae	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Add+0x7e	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:193
#	0x15a54b6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0x186	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:904
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 1966080 [15: 29491200] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1: 1810432 [3: 5431296] @ 0x40a135 0x40a2f3 0x40d23d 0x15fe78d 0x15fb5b1 0x45d051
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addDropped+0x1c4c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:567
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1c4c		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:864
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2431: 1711424 [3603: 2536512] @ 0x41156b 0x4be489 0xc689b9 0xc5f419 0xe8b90c 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0x41156a	reflect.unsafe_New+0x2a														/root/working/go/src/runtime/malloc.go:845
#	0x4be488	reflect.New+0x48														/root/working/go/src/reflect/value.go:2150
#	0xc689b8	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).New+0x288					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:278
#	0xc5f418	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.UseOrCreateObject+0x78					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:111
#	0xe8b90b	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x66b		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:153
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

1: 1622016 [18: 29196288] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

3: 1523712 [39: 19808256] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

2289: 1464960 [3322: 2126080] @ 0x40d427 0x1137673 0x10d7b22 0x10cf4af 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0x1137672	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decResourceList+0x612			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71970
#	0x10d7b21	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ResourceList).CodecDecodeSelf+0x41			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:49839
#	0x10cf4ae	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x72e			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47500
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

2227: 1425280 [3240: 2073600] @ 0x40d427 0x1137673 0x10d7b22 0x10cf340 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0x1137672	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decResourceList+0x612			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71970
#	0x10d7b21	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ResourceList).CodecDecodeSelf+0x41			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:49839
#	0x10cf33f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x5bf			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47493
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

10955: 1402240 [16063: 2056064] @ 0xe65385 0xde7cc0 0xde747c 0x10d44aa 0x10d4165 0x1134646 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xe65384	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString+0x1d4				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go:851
#	0xde7cbf	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV+0x7df				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17926
#	0xde747b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringX+0x5b				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17823
#	0x10d44a9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap+0x2e9		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48942
#	0x10d4164	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).CodecDecodeSelf+0x124			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48896
#	0x1134645	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x905		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71488
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

3: 1376256 [13: 5963776] @ 0x40a135 0x40a2f3 0x40d23d 0x15fd18d 0x15fb5b1 0x45d051
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:562
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:903
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 1310720 [1: 1310720] @ 0x15a5e01 0x15a077d 0x15984b1 0x15c05e2 0x171c28b 0x7398b7 0x45d051
#	0x15a5e00	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.newStripeSeries+0x30			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:986
#	0x15a077c	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.NewHead+0x8c				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:188
#	0x15984b0	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.Open+0x470				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:216
#	0x15c05e1	github.com/prometheus/prometheus/storage/tsdb.Open+0x2a1						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:143
#	0x171c28a	main.main.func18+0x21a											/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:500
#	0x7398b6	github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1+0x26	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38

2: 1245184 [18: 11206656] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1: 1089536 [14: 15253504] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1: 1073152 [21797: 23391494144] @ 0x15f34e4 0x15fb424 0x45d051
#	0x15f34e3	github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get+0xd3		/root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go:59
#	0x15fb423	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x503	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:661

9: 1032192 [82: 9404416] @ 0x40a135 0x40a2f3 0x40d23d 0x15fe78d 0x15fb5b1 0x45d051
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addDropped+0x1c4c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:567
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1c4c		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:864
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

974: 997376 [4491: 4598784] @ 0x11346fd 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0x11346fc	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x9bc		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71479
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

30: 983040 [85: 2785280] @ 0x624710 0x627a5e 0x6270ca 0x626f08 0x15fa04d 0x15fb4dd 0x45d051
#	0x62470f	compress/flate.(*dictDecoder).init+0x27f					/root/working/go/src/compress/flate/dict_decoder.go:43
#	0x62470f	compress/flate.NewReader+0x27f							/root/working/go/src/compress/flate/inflate.go:793
#	0x627a5d	compress/gzip.(*Reader).readHeader+0x30d					/root/working/go/src/compress/gzip/gunzip.go:238
#	0x6270c9	compress/gzip.(*Reader).Reset+0x159						/root/working/go/src/compress/gzip/gunzip.go:113
#	0x626f07	compress/gzip.NewReader+0x57							/root/working/go/src/compress/gzip/gunzip.go:94
#	0x15fa04c	github.com/prometheus/prometheus/retrieval.(*targetScraper).scrape+0x47c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:448
#	0x15fb4dc	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x5bc		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:664

1: 983040 [3: 2949120] @ 0x1588d54 0x1588aaf 0x15a54b7 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x1588d53	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).addFor+0x213	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:201
#	0x1588aae	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Add+0x7e	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:193
#	0x15a54b6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0x186	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:904
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

17: 974848 [93: 5332992] @ 0x40a135 0x40a2f3 0x40d23d 0x15fe78d 0x15fb5b1 0x45d051
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addDropped+0x1c4c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:567
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1c4c		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:864
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

3: 786432 [3: 786432] @ 0x7e5f38 0x7e6574 0x7e5699 0x7e520c 0x7e6fd7 0xcb3e4c 0x45d051
#	0x7e5f37	bufio.NewWriterSize+0x117										/root/working/go/src/bufio/bufio.go:541
#	0x7e5f37	github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*syncBuffer).rotateFile+0x117		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:838
#	0x7e6573	github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*loggingT).createFiles+0xf3		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:867
#	0x7e5698	github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*loggingT).output+0x448			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:686
#	0x7e520b	github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*loggingT).printf+0x14b			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:655
#	0x7e6fd6	github.com/prometheus/prometheus/vendor/github.com/golang/glog.Errorf+0x66				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:1118
#	0xcb3e4b	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x22b	/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:109

1: 745472 [13: 9691136] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1: 630784 [60: 37847040] @ 0x40a135 0x40a2f3 0x40d23d 0x15b8e71 0x1588982 0x15a3a4b 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x15b8e70	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc.func1+0x90		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:631
#	0x1588981	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Iter+0x1c1	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:181
#	0x15a3a4a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x20a			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:629
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1: 614400 [51: 31334400] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

2001: 576288 [2944: 847872] @ 0x40d427 0xdff40b 0xdfef7e 0xd5eb9c 0xd5963b 0xd5eabc 0xe5c544 0x10d8a24 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xdff40a	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecMapStringStringV+0x31a				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:21140
#	0xdfef7d	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*decFnInfo).fastpathDecMapStringStringR+0xad		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:21079
#	0xd5eb9b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decodeValue+0x9b					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1629
#	0xd5963a	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*decFnInfo).kStruct+0x8da					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:754
#	0xd5eabb	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decodeI+0x18b					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1620
#	0xe5c543	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.genHelperDecoder.DecFallback+0x53				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/gen-helper.generated.go:150
#	0x10d8a23	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x583			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50086
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

3877: 558288 [5705: 821520] @ 0xe65385 0xde7cc0 0xde747c 0x10d44aa 0x10d4165 0x1134646 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xe65384	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString+0x1d4				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go:851
#	0xde7cbf	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV+0x7df				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17926
#	0xde747b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringX+0x5b				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17823
#	0x10d44a9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap+0x2e9		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48942
#	0x10d4164	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).CodecDecodeSelf+0x124			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48896
#	0x1134645	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x905		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71488
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

7981: 510784 [11628: 744192] @ 0xde7eba 0xde747c 0x10d44aa 0x10d4165 0x1134646 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xde7eb9	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV+0x9d9				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17911
#	0xde747b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringX+0x5b				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17823
#	0x10d44a9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap+0x2e9		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48942
#	0x10d4164	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).CodecDecodeSelf+0x124			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48896
#	0x1134645	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x905		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71488
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

1: 491520 [14: 6881280] @ 0x1588d54 0x1588aaf 0x15a54b7 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x1588d53	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).addFor+0x213	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:201
#	0x1588aae	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Add+0x7e	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:193
#	0x15a54b6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0x186	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:904
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 393216 [7140: 1403781120] @ 0x15a2d1d 0x15be1ec 0x15c135e 0x15c20f8 0x156b695 0x16019ac 0x15fe852 0x15fb5b1 0x45d051
#	0x15a2d1c	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).AddFast+0x1bc	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:515
#	0x15be1eb	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).AddFast+0x5b		<autogenerated>:1
#	0x15c135d	github.com/prometheus/prometheus/storage/tsdb.appender.AddFast+0x5d					/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:243
#	0x15c20f7	github.com/prometheus/prometheus/storage/tsdb.(*appender).AddFast+0x97					<autogenerated>:1
#	0x156b694	github.com/prometheus/prometheus/storage.(*fanoutAppender).AddFast+0x84					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:148
#	0x16019ab	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).AddFast+0xab				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:249
#	0x15fe851	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1d11					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:820
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 393216 [13: 5111808] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

2: 376832 [6: 1130496] @ 0x40a135 0x40a2f3 0x40d23d 0x11ddb31 0x11df014 0x11e2cf3 0x11e2182 0x11ead33 0xcb387e 0xcb3641 0xcb358d 0x45d051
#	0x11ddb30	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).queueActionLocked+0x210	/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:334
#	0x11df013	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Replace+0x233			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:477
#	0x11e2cf2	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*Reflector).syncWith+0x1c2		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/reflector.go:336
#	0x11e2181	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0x721		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/reflector.go:258
#	0x11ead32	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*Reflector).RunUntil.func1+0x32		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/reflector.go:200
#	0xcb387d	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96
#	0xcb3640	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xa0			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97
#	0xcb358c	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52

13: 372736 [77: 2207744] @ 0x40a135 0x40a2f3 0x40d23d 0x15fe78d 0x15fb5b1 0x45d051
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addDropped+0x1c4c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:567
#	0x15fe78c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1c4c		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:864
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

3: 368640 [11976: 1471610880] @ 0x15f34e4 0x15fb424 0x45d051
#	0x15f34e3	github.com/prometheus/prometheus/pkg/pool.(*BytesPool).Get+0xd3		/root/gopath/src/github.com/prometheus/prometheus/pkg/pool/pool.go:59
#	0x15fb423	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x503	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:661

1: 286720 [3: 860160] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

3451: 276080 [5112: 408960] @ 0xe65385 0xde7cc0 0xde747c 0x10d44aa 0x10d4165 0x1134646 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xe65384	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString+0x1d4				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go:851
#	0xde7cbf	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV+0x7df				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17926
#	0xde747b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringX+0x5b				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17823
#	0x10d44a9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap+0x2e9		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48942
#	0x10d4164	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).CodecDecodeSelf+0x124			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48896
#	0x1134645	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x905		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71488
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

1: 262144 [1: 262144] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

6: 245760 [41: 1679360] @ 0x40a135 0x40a2f3 0x40c962 0x15fe8e8 0x15fb5b1 0x45d051
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x1da7	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1da7		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:823
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

6: 245760 [75: 3072000] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

3: 245760 [45: 3686400] @ 0x40a135 0x40a2f3 0x40c962 0x15fe8e8 0x15fb5b1 0x45d051
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x1da7	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fe8e7	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1da7		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:823
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 245760 [16: 1966080] @ 0x1588d54 0x1588aaf 0x15a54b7 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x1588d53	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).addFor+0x213	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:201
#	0x1588aae	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Add+0x7e	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:193
#	0x15a54b6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0x186	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:904
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

3: 221184 [8: 589824] @ 0x1588d54 0x1588aaf 0x15a54b7 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x1588d53	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).addFor+0x213	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:201
#	0x1588aae	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Add+0x7e	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:193
#	0x15a54b6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0x186	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:904
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

3348: 214272 [4984: 318976] @ 0xe65385 0xde7cc0 0xde747c 0x10d44aa 0x10d4165 0x1134646 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xe65384	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString+0x1d4				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go:851
#	0xde7cbf	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV+0x7df				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17926
#	0xde747b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringX+0x5b				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17823
#	0x10d44a9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap+0x2e9		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48942
#	0x10d4164	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).CodecDecodeSelf+0x124			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48896
#	0x1134645	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x905		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71488
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

3: 196608 [6381: 418185216] @ 0x15b0f77 0x15b21a7 0x15a2f71 0x1599795 0x15be2ee 0x15c1591 0x15c2195 0x156b7f9 0x1604a3c 0x15fd647 0x15fb5b1 0x45d051
#	0x15b0f76	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).getBuffer+0x86	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:309
#	0x15b21a6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).LogSeries+0x46	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:439
#	0x15a2f70	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Commit+0xa0	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:527
#	0x1599794	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.dbAppender.Commit+0x34	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:318
#	0x15be2ed	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Commit+0x4d	<autogenerated>:1
#	0x15c1590	github.com/prometheus/prometheus/storage/tsdb.appender.Commit+0x30				/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:258
#	0x15c2194	github.com/prometheus/prometheus/storage/tsdb.(*appender).Commit+0x44				<autogenerated>:1
#	0x156b7f8	github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit+0x48				/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:161
#	0x1604a3b	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Commit+0x3b			<autogenerated>:1
#	0x15fd646	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0xb06				/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:940
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690				/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 196608 [5: 491520] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1558: 174496 [2285: 255920] @ 0xe65385 0xde7cc0 0xde747c 0x10d44aa 0x10d4165 0x1134646 0x10cf01b 0x10ced25 0x10d8b19 0x10d8445 0xd5e682 0xd5cefb 0xe8b9bf 0xe8e8b7 0xe8eaaf 0xc5f357 0x11c7f69 0xcb3d58 0x45d051
#	0xe65384	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString+0x1d4				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/json.go:851
#	0xde7cbf	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringV+0x7df				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17926
#	0xde747b	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.fastpathT.DecSliceStringX+0x5b				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/fast-path.generated.go:17823
#	0x10d44a9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).codecDecodeSelfFromMap+0x2e9		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48942
#	0x10d4164	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*ContainerImage).CodecDecodeSelf+0x124			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:48896
#	0x1134645	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainerImage+0x905		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:71488
#	0x10cf01a	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).codecDecodeSelfFromMap+0x29a			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47556
#	0x10ced24	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*NodeStatus).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:47451
#	0x10d8b18	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).codecDecodeSelfFromMap+0x678			/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50101
#	0x10d8444	github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.(*Node).CodecDecodeSelf+0x124				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1/types.generated.go:50015
#	0xd5e681	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).decode+0x11e1					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1536
#	0xd5cefa	github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*Decoder).Decode+0x7a					/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec/decode.go:1383
#	0xe8b9be	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode+0x71e		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:158
#	0xe8e8b6	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.DirectDecoder.Decode+0x96		/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266
#	0xe8eaae	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.(*DirectDecoder).Decode+0x9e	<autogenerated>:1
#	0xc5f356	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime.Decode+0x76						/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:54
#	0x11c7f68	github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x1b8					/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/rest/watch/decoder.go:62
#	0xcb3d57	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137				/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

1: 155648 [4: 622592] @ 0x40a135 0x40a2f3 0x40d23d 0x11e8325 0x11e77da 0x11e5988 0x11eb8de 0x11dec68 0x11dc540 0x11eb7ea 0xcb387e 0xcb3641 0xcb358d 0x11dc39e 0x11e4a30 0x45d051
#	0x11e8324	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*threadSafeMap).Add+0xd4											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/thread_safe_store.go:71
#	0x11e77d9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*cache).Add+0xf9												/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/store.go:128
#	0x11e5987	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas+0x1f7									/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:352
#	0x11eb8dd	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas-fm+0x3d								/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:198
#	0x11dec67	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x267											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451
#	0x11dc53f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:147
#	0x11eb7e9	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29	/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121
#	0xcb387d	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96
#	0xcb3640	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xa0											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97
#	0xcb358c	github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c												/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52
#	0x11dc39d	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x22d											/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/controller.go:121
#	0x11e4a2f	github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x37f										/root/gopath/src/github.com/prometheus/prometheus/vendor/k8s.io/client-go/tools/cache/shared_informer.go:213

1: 155648 [4: 622592] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

1: 155648 [6542: 1018249216] @ 0x15a2d1d 0x15be1ec 0x15c135e 0x15c20f8 0x156b695 0x16019ac 0x15fe852 0x15fb5b1 0x45d051
#	0x15a2d1c	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).AddFast+0x1bc	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:515
#	0x15be1eb	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).AddFast+0x5b		<autogenerated>:1
#	0x15c135d	github.com/prometheus/prometheus/storage/tsdb.appender.AddFast+0x5d					/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:243
#	0x15c20f7	github.com/prometheus/prometheus/storage/tsdb.(*appender).AddFast+0x97					<autogenerated>:1
#	0x156b694	github.com/prometheus/prometheus/storage.(*fanoutAppender).AddFast+0x84					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:148
#	0x16019ab	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).AddFast+0xab				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:249
#	0x15fe851	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1d11					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:820
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

3: 122880 [43: 1761280] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 114688 [16: 917504] @ 0x1588d54 0x1588aaf 0x15a54b7 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x1588d53	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).addFor+0x213	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:201
#	0x1588aae	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Add+0x7e	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:193
#	0x15a54b6	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0x186	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:904
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 114688 [50: 2867200] @ 0x40a135 0x40a2f3 0x40d23d 0x15fd18d 0x15fb5b1 0x45d051
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:562
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:903
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 98304 [5297: 520716288] @ 0x15a2d1d 0x15be1ec 0x15c135e 0x15c20f8 0x156b695 0x16019ac 0x15fe852 0x15fb5b1 0x45d051
#	0x15a2d1c	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).AddFast+0x1bc	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:515
#	0x15be1eb	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).AddFast+0x5b		<autogenerated>:1
#	0x15c135d	github.com/prometheus/prometheus/storage/tsdb.appender.AddFast+0x5d					/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:243
#	0x15c20f7	github.com/prometheus/prometheus/storage/tsdb.(*appender).AddFast+0x97					<autogenerated>:1
#	0x156b694	github.com/prometheus/prometheus/storage.(*fanoutAppender).AddFast+0x84					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:148
#	0x16019ab	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).AddFast+0xab				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:249
#	0x15fe851	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1d11					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:820
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

4: 98304 [4: 98304] @ 0x777b4c 0x7777d7 0x7773c5 0x77731e 0x787d89 0x78dcfb 0x45d051
#	0x777b4b	github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile.(*stream).merge+0x2ab				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile/stream.go:222
#	0x7777d6	github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile.(*Stream).flush+0x56				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile/stream.go:180
#	0x7773c4	github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile.(*Stream).insert+0x94				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile/stream.go:121
#	0x77731d	github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile.(*Stream).Insert+0x4d				/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/beorn7/perks/quantile/stream.go:114
#	0x787d88	github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.(*summary).flushColdBuf+0x58	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/summary.go:361
#	0x78dcfa	github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.(*summary).asyncFlush.func1+0x2a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/summary.go:339

2: 98304 [10: 491520] @ 0x158838b 0x15a3981 0x15a1649 0x159b389 0x1599ab9 0x1598fba 0x45d051
#	0x158838a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index.(*MemPostings).Delete+0x30a	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/index/postings.go:159
#	0x15a3980	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).gc+0x140			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:623
#	0x15a1648	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Truncate+0xb8		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:347
#	0x159b388	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).reload+0xbf8			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:526
#	0x1599ab8	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact+0x278			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:367
#	0x1598fb9	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run+0x309			/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:263

206: 92288 [1234: 552832] @ 0x155fecf 0x1561c8e 0x15616d8 0x15f946e 0x1603311 0x15fd05d 0x15fb5b1 0x45d051
#	0x155fece	github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels+0x6e			/root/gopath/src/github.com/prometheus/prometheus/pkg/labels/labels.go:243
#	0x1561c8d	github.com/prometheus/prometheus/pkg/relabel.relabel+0x53d				/root/gopath/src/github.com/prometheus/prometheus/pkg/relabel/relabel.go:102
#	0x15616d7	github.com/prometheus/prometheus/pkg/relabel.Process+0x87				/root/gopath/src/github.com/prometheus/prometheus/pkg/relabel/relabel.go:33
#	0x15f946d	github.com/prometheus/prometheus/retrieval.(*scrapePool).mutateSampleLabels+0x2ed	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:352
#	0x1603310	github.com/prometheus/prometheus/retrieval.newScrapePool.func1.1+0x50			/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:169
#	0x15fd05c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x51c			/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:860
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690			/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

211: 87776 [712: 296192] @ 0x155fecf 0x1561c8e 0x15616d8 0x15f946e 0x1603311 0x15fd05d 0x15fb5b1 0x45d051
#	0x155fece	github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels+0x6e			/root/gopath/src/github.com/prometheus/prometheus/pkg/labels/labels.go:243
#	0x1561c8d	github.com/prometheus/prometheus/pkg/relabel.relabel+0x53d				/root/gopath/src/github.com/prometheus/prometheus/pkg/relabel/relabel.go:102
#	0x15616d7	github.com/prometheus/prometheus/pkg/relabel.Process+0x87				/root/gopath/src/github.com/prometheus/prometheus/pkg/relabel/relabel.go:33
#	0x15f946d	github.com/prometheus/prometheus/retrieval.(*scrapePool).mutateSampleLabels+0x2ed	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:352
#	0x1603310	github.com/prometheus/prometheus/retrieval.newScrapePool.func1.1+0x50			/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:169
#	0x15fd05c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x51c			/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:860
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690			/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

3: 86016 [32: 917504] @ 0x40a135 0x40a2f3 0x40d23d 0x15fd18d 0x15fb5b1 0x45d051
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeCache).addRef+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:562
#	0x15fd18c	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x64c	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:903
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

36: 82944 [36: 82944] @ 0x40a135 0x40a2f3 0x40c962 0x15a5a55 0x15a6505 0x15a5418 0x15a5305 0x15a2968 0x15be13b 0x15c1098 0x15c1ffb 0x156b4cb 0x160185b 0x15fd0d2 0x15fb5b1 0x45d051
#	0x15a5a54	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.seriesHashmap.set+0x1b4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:947
#	0x15a6504	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*stripeSeries).getOrSet+0xd4	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:1077
#	0x15a5417	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreateWithID+0xe7	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:896
#	0x15a5304	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).getOrCreate+0xc4		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:890
#	0x15a2967	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).Add+0xa7		/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:485
#	0x15be13a	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add+0x7a		<autogenerated>:1
#	0x15c1097	github.com/prometheus/prometheus/storage/tsdb.appender.Add+0xa7						/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:227
#	0x15c1ffa	github.com/prometheus/prometheus/storage/tsdb.(*appender).Add+0x8a					<autogenerated>:1
#	0x156b4ca	github.com/prometheus/prometheus/storage.(*fanoutAppender).Add+0x7a					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:134
#	0x160185a	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add+0x7a				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:238
#	0x15fd0d1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x591					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:869
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

1: 81920 [66: 5406720] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

4: 76288 [34: 648448] @ 0x40a135 0x40a2f3 0x40c962 0x15fd2c2 0x15fb5b1 0x45d051
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeCache).trackStaleness+0x781	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:579
#	0x15fd2c1	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x781		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:901
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690		/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

144: 69120 [361: 173280] @ 0x15f4d6d 0x15fcfd7 0x15fb5b1 0x45d051
#	0x15f4d6c	github.com/prometheus/prometheus/pkg/textparse.(*Parser).Metric+0x8c	/root/gopath/src/github.com/prometheus/prometheus/pkg/textparse/parse.go:114
#	0x15fcfd6	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x496	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:855
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

151: 67648 [345: 154560] @ 0x15f4d6d 0x15fcfd7 0x15fb5b1 0x45d051
#	0x15f4d6c	github.com/prometheus/prometheus/pkg/textparse.(*Parser).Metric+0x8c	/root/gopath/src/github.com/prometheus/prometheus/pkg/textparse/parse.go:114
#	0x15fcfd6	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x496	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:855
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690	/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

2: 65536 [2173: 71204864] @ 0x15a2d1d 0x15be1ec 0x15c135e 0x15c20f8 0x156b695 0x16019ac 0x15fe852 0x15fb5b1 0x45d051
#	0x15a2d1c	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*headAppender).AddFast+0x1bc	/root/gopath/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:515
#	0x15be1eb	github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).AddFast+0x5b		<autogenerated>:1
#	0x15c135d	github.com/prometheus/prometheus/storage/tsdb.appender.AddFast+0x5d					/root/gopath/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:243
#	0x15c20f7	github.com/prometheus/prometheus/storage/tsdb.(*appender).AddFast+0x97					<autogenerated>:1
#	0x156b694	github.com/prometheus/prometheus/storage.(*fanoutAppender).AddFast+0x84					/root/gopath/src/github.com/prometheus/prometheus/storage/fanout.go:148
#	0x16019ab	github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).AddFast+0xab				/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:249
#	0x15fe851	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append+0x1d11					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:820
#	0x15fb5b0	github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run+0x690					/root/gopath/src/github.com/prometheus/prometheus/retrieval/scrape.go:684

brancz · May 17, 2018

Can you confirm, that when you do kubectl get pod <your-prometheus-pod> -oyaml, that it actually says that your container has been OOMKilled?

In my case the graph of the container usage looks similar but I think we're just seeing mmap memory being freed.

Concretely I've been running a setup where the container_memory_usage_bytes looks similar to yours. My Prometheus container has a memory limit of 1Gi:

However heap usage (reported by the go runtime) seems stable and not ever growing:

Also the process has been running for >24h, so besides Kubernetes not reporting an OOMKill to me, the process itself is also confirming that.

Could you try these queries and validate this against your setup?

piaoyu · May 18, 2018

@brancz

container_memory_usage_bytes not the real memory but container_memory_rss does

krasi-georgiev · May 24, 2018

the screenshot doesn't seem to show anything useful.

@piaoyu did you check the logs to see if Prometheus really gets OOM killed?

piaoyu · May 28, 2018

@krasi-georgiev I check the logs it really gets OOM killed.

grantstephens · May 28, 2018

I've also experienced the same problem and it has stopped when I removed the k8 cluster from the config.

cofyc · May 28, 2018

I'm looking into this, and trying to reproduce this issue in my cluster.

/prometheus $ prometheus --version
 prometheus, version 2.2.1 (branch: HEAD, revision: 94e4a4321761f83207ad11542ee971e7d5220e80)
  build user:       root@sz-bcs-a151
  build date:       20180508-12:56:09
  go version:       go1.9.4

@piaoyu From output of prometheus --version, you are using a custom build, could you try to use quay.io/prometheus/prometheus:v2.2.1 instead? If it still get OOM killed, let me know.

quay.io/prometheus/prometheus:v2.2.1 revision is bc6058c81272a8d938c05e75607371284236aadc which does not contain k8s sd refactoring.

cofyc · May 28, 2018

@grantstephens Could you provide output of prometheus --version?

grantstephens · May 28, 2018

prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

Let me know if you need any more information.

djsly · Jun 1, 2018

Hello, we are also affected on three different clusters, we are getting POD OOM killed every few hours.
What could be beneficial for you to help with the investigation ?

simonpasquier · Jun 4, 2018

@djsly it would helpful if you could share:

The exact Prometheus version.
System logs about the OOM events.
Prometheus logs.
graphs of the following metrics
- container_memory_usage_bytes for the Prometheus pod(s).
- prometheus_tsdb_head_series
- rate(prometheus_tsdb_head_series_created_total[1m])

krasi-georgiev · Jun 6, 2018

for everyone in this issue I just realised that if you are testing the binary from the release page this doesn't include the refactoring so make sure you build a new one from master or the 2.2 branch.

https://github.com/prometheus/prometheus/tree/release-2.2

I can build the binary for anyone willing to test it.

piaoyu · Jun 7, 2018

@cofyc quay.io/prometheus/prometheus:v2.2.1 seems not oom problem

krasi-georgiev · Jun 7, 2018

@piaoyu did you read my comment above?
quay.io/prometheus/prometheus:v2.2.1 is before the refactoring which is suppose to fix the problem

krasi-georgiev · Jun 7, 2018

v2.3.0 is out so try that.

quay.io/prometheus/prometheus:v2.3.0

piaoyu · Jun 12, 2018

@krasi-georgiev OK . I will try quay.io/prometheus/prometheus:v2.3.0

dfredell · Jun 18, 2018

My prometheus was sucking up a ton of memory and disk. More than I expected. I had storage.tsdb.retention set to 10 days, and gave the prometheus docker 50G disk and 4G of memory. It wasn't happy. I tried upgrading to 2.3.0 and I think that made things even worse.
Doing more research I found https://www.robustperception.io/which-are-my-biggest-metrics/ which pointed out that I had a custom metric with 166070 entries. I think I will fix my metrics and see if that fixes my prometheus stability issue. Adding it here in case others have this issue too.

topk(10, count by (__name__, job)({__name__=~".+"}))

Element	Value
containerpilot_events{job="consul"}	166070
consul_catalog_service_node_healthy{job="consul"}	150
consul_health_service_status{job="consul"}	141
go_gc_duration_seconds{job="consul"}	105

ex
containerpilot_events{job="consul"}

Element	Value
containerpilot_events{code="Metric",instance="192.168.130.252:9090",job="consul",service="containerpilot",source="tb_process_load\|0.000024106811388486265"}	1
containerpilot_events{code="Metric",instance="192.168.130.252:9090",job="consul",service="containerpilot",source="tb_process_load\|0.000024114556647921814"}	1
containerpilot_events{code="Metric",instance="192.168.130.252:9090",job="consul",service="containerpilot",source="tb_process_load\|0.000024234685599017417"}	1
containerpilot_events{code="Metric",instance="192.168.130.252:9090",job="consul",service="containerpilot",source="tb_process_load\|0.000024337671686951137"}	1

krasi-georgiev · Jun 18, 2018

@dfredell thanks for the report, but can you open a new issue as this one is getting to long.

@piaoyu can you please confirm if 2.3 fixed the problem for you so we can close this one?

piaoyu · Jun 22, 2018

@krasi-georgiev seem like 2.3 fixed the problem for 12hours memory watch.

krasi-georgiev · Jun 22, 2018

Nice , closing this as resolved!
Feel free to reopen or file a new one if you notice any other problems.

Thanks for the team work 👍

piaoyu referenced this issue May 15, 2018
Closed
Prometheus version 2.0 memory leak ? #3685

brian-brazil added the kind/more-info-needed label Jun 13, 2018

krasi-georgiev closed this Jun 22, 2018

prometheus/prometheus

Join GitHub today

Prometheus Release 2.2.0 Memory Leak ? #4164

Comments

piaoyu commented May 15, 2018 • edited

Proposal

Bug Report

This comment has been minimized.

piaoyu commented May 15, 2018

This comment has been minimized.

piaoyu commented May 15, 2018

This comment has been minimized.

piaoyu commented May 15, 2018

piaoyu referenced this issue May 15, 2018

Prometheus version 2.0 memory leak ? #3685

This comment has been minimized.

brancz commented May 15, 2018

This comment has been minimized.

piaoyu commented May 15, 2018 • edited

This comment has been minimized.

brancz commented May 15, 2018

This comment has been minimized.

krasi-georgiev commented May 15, 2018

This comment has been minimized.

cofyc commented May 16, 2018 • edited

This comment has been minimized.

piaoyu commented May 17, 2018 • edited

This comment has been minimized.

piaoyu commented May 17, 2018 • edited

This comment has been minimized.

piaoyu commented May 17, 2018

This comment has been minimized.

brancz commented May 17, 2018

This comment has been minimized.

piaoyu commented May 18, 2018

This comment has been minimized.

krasi-georgiev commented May 24, 2018

This comment has been minimized.

piaoyu commented May 28, 2018

This comment has been minimized.

grantstephens commented May 28, 2018

This comment has been minimized.

cofyc commented May 28, 2018 • edited

This comment has been minimized.

cofyc commented May 28, 2018

This comment has been minimized.

grantstephens commented May 28, 2018

This comment has been minimized.

djsly commented Jun 1, 2018

This comment has been minimized.

simonpasquier commented Jun 4, 2018

This comment has been minimized.

krasi-georgiev commented Jun 6, 2018

This comment has been minimized.

piaoyu commented Jun 7, 2018

This comment has been minimized.

krasi-georgiev commented Jun 7, 2018

This comment has been minimized.

krasi-georgiev commented Jun 7, 2018

This comment has been minimized.

piaoyu commented Jun 12, 2018

brian-brazil added the kind/more-info-needed label Jun 13, 2018

This comment has been minimized.

dfredell commented Jun 18, 2018

This comment has been minimized.

krasi-georgiev commented Jun 18, 2018 • edited

This comment has been minimized.

piaoyu commented Jun 22, 2018

This comment has been minimized.

krasi-georgiev commented Jun 22, 2018

krasi-georgiev closed this Jun 22, 2018

piaoyu commented May 15, 2018 •

edited

piaoyu commented May 15, 2018 •

edited

cofyc commented May 16, 2018 •

edited

piaoyu commented May 17, 2018 •

edited

piaoyu commented May 17, 2018 •

edited

cofyc commented May 28, 2018 •

edited

krasi-georgiev commented Jun 18, 2018 •

edited