Prometheus abnormal crash

[mtoulon]

After Promethues has been running for several hours I observed an error in 'go kernel' I am running Promethues 2.6.0 below is the error extract.

runtime: nelems=102 nalloc=2 previous allocCount=1 nfreed=65535
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0x1c458e6, 0x20)
/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.(*mspan).sweep(0x7fa08293d8e8, 0x100433d00, 0xc000000c00)
/usr/local/go/src/runtime/mgcsweep.go:298 +0x879
runtime.sweepone(0x4344a7)
/usr/local/go/src/runtime/mgcsweep.go:113 +0x10f
runtime.gosweepone.func1()
/usr/local/go/src/runtime/mgcsweep.go:137 +0x2b
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
/usr/local/go/src/runtime/proc.go:122

Bug Report

What did you do?
The system was using all core' allocated so I extended from 2 cores to 4 cores

What did you expect to see?
Improved performance, and no alerts on CPU limit reached

What did you see instead? Under which circumstances?
The prometheus crashed

Environment
Open Shift Container Platform
openshift v3.9.51
kubernetes v1.9.1+a0ce1bc657

• System information:

/etc/prometheus # uname -srm
Linux 3.10.0-957.5.1.el7.x86_64 x86_64
/etc/prometheus #

• Prometheus version:

/etc/prometheus # prometheus --version
prometheus, version 2.6.0 (branch: HEAD, revision: dbd1d58)
build user: root@bf5760470f13
build date: 20181217-15:14:46
go version: go1.11.3
/etc/prometheus #

• Alertmanager version:

  insert output of alertmanager --version here (if relevant to the issue)

• Prometheus configuration file:

/etc/prometheus # cat /etc/prometheus/prometheus-kubernetes17.yml
#THIS FOR KUBERNETES 1.7 (OCP 3.7) AND 1.8 (OCP 3.8) AND 1.9 (OCP 3.9)

global:
  scrape_interval:     30s
  evaluation_interval: 30s

rule_files:
  - "/etc/prometheus-rules/*.rules"


########################### DISCOVERY ###########################
# https://raw.githubusercontent.com/prometheus/prometheus/master/documentation/examples/prometheus-kubernetes.yml
# A scrape configuration for running Prometheus on a Kubernetes cluster.
# This uses separate scrape configs for cluster components (i.e. API server, node)
# and services to allow each to use different authentication configs.
#
# Kubernetes labels will be added as Prometheus labels on metrics via the
# `labelmap` relabeling action.
#
# If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
# for the kubernetes-cadvisor job; you will need to edit or remove this job.

# Scrape config for API servers.
#
# Kubernetes exposes API servers as endpoints to the default/kubernetes
# service so this uses `endpoints` role and uses relabelling to only keep
# the endpoints associated with the default/kubernetes service using the
# default named port `https`. This works for single API server deployments as
# well as HA API server deployments.

scrape_configs:
- job_name: 'kubernetes-apiservers'

  kubernetes_sd_configs:
  - role: endpoints

  # Default to scraping over https. If required, just disable this or change to
  # `http`.
  scheme: https

  # This TLS & bearer token file config is used to connect to the actual scrape
  # endpoints for cluster components. This is separate to discovery auth
  # configuration because discovery & scraping are two separate concerns in
  # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  # the cluster. Otherwise, more config options have to be provided within the
  # <kubernetes_sd_config>.
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    # If your node certificates are self-signed or use a different CA to the
    # master CA, then disable certificate verification below. Note that
    # certificate verification is an integral part of a secure infrastructure
    # so this should only be disabled in a controlled environment. You can
    # disable certificate verification by uncommenting the line below.
    #
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  # Keep only the default/kubernetes service endpoints for the https port. This
  # will add targets for each API server which Kubernetes adds an endpoint to
  # the default/kubernetes service.
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

# Scrape config for nodes (kubelet).
- job_name: 'kubernetes-nodes'
  scrape_interval: 60s
  scrape_timeout: 60s

  # Default to scraping over https. If required, just disable this or change to
  # `http`.
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  kubernetes_sd_configs:
  - role: node

  # Drop a very high cardinality metric that is incorrect in 3.7. It will be
  # fixed in 3.9.
  metric_relabel_configs:
  - source_labels: [__name__]
    action: drop
    regex: 'openshift_sdn_pod_(setup|teardown)_latency(.*)'
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

  #https://stackoverflow.com/questions/44718268/how-to-monitor-disk-usage-of-kubernetes-persistent-volumes?noredirect=1&lq=1
  # - target_label: __address__
  #   replacement: kubernetes.default.svc:443
  # - source_labels: [__meta_kubernetes_node_name]
  #   regex: (.+)
  #   target_label: __metrics_path__
  #   replacement: /api/v1/nodes/${1}/proxy/metrics

# Scrape kubelet container metrics
- job_name: 'kubernetes-cadvisor'
  scrape_interval: 60s
  scrape_timeout: 60s

  # Default to scraping over https. If required, just disable this or change to
  # `http`.
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  metrics_path: /metrics/cadvisor
  kubernetes_sd_configs:
  - role: node
  # Exclude a set of high cardinality metrics that can contribute to significant
  # memory use in large clusters. These can be selectively enabled as necessary
  # for medium or small clusters.
  metric_relabel_configs:
  - source_labels: [__name__]
    action: drop
  #  regex: 'container_(cpu_user_seconds_total|cpu_cfs_periods_total|memory_usage_bytes|memory_swap|memory_cache|last_seen|fs_(read_seconds_total|write_seconds_total|sector_(.*)|io_(.*)|reads_merged_total|writes_merged_total)|tasks_state|memory_failcnt|memory_failures_total|spec_memory_swap_limit_bytes|fs_(.*)_bytes_total|spec_(.*))'
    regex: 'container_(cpu_cfs_periods_total|memory_cache|last_seen|fs_(read_seconds_total|write_seconds_total|sector_(.*)|io_(.*)|reads_merged_total|writes_merged_total)|tasks_state|memory_failcnt|memory_failures_total|spec_memory_swap_limit_bytes|fs_(.*)_bytes_total|spec_(.*))'

  #AL only consider the current namespace + no namespace label (when cpu/mem ... on node)
  - source_labels: [namespace]
    regex: 'prod-eaa|'
    action: keep

  #AL to have pod and container with same labels as in kubestate
  - source_labels: [pod_name]
    target_label: pod
  - source_labels: [container_name]
    target_label: container
  - regex: '(pod_name|container_name)'
    action: labeldrop
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

##AL NOT USED TODAY pods should be scrap flag to put to true
# Scrape config for service endpoints.
#
# The relabeling allows the actual service scrape endpoint to be configured
# via the following annotations:
#
# * `prometheus.io/scrape`: Only scrape services that have a value of `true`
# * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
# to set this to `https` & most likely set the `tls_config` of the scrape config.
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
# * `prometheus.io/port`: If the metrics are exposed on a different port to the
# service then set this appropriately.
#- job_name: 'kubernetes-service-endpoints'
#
#  kubernetes_sd_configs:
#  - role: endpoints
#
#  relabel_configs:
#  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
#    action: keep
#    regex: true
#  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
#    action: replace
#    target_label: __scheme__
#    regex: (https?)
#  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
#    action: replace
#    target_label: __metrics_path__
#    regex: (.+)
#  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
#    action: replace
#    target_label: __address__
#    regex: (.+)(?::\d+);(\d+)
#    replacement: $1:$2
#  - action: labelmap
#    regex: __meta_kubernetes_service_label_(.+)
#  - source_labels: [__meta_kubernetes_service_namespace]
#    action: replace
#    target_label: kubernetes_namespace
#  - source_labels: [__meta_kubernetes_service_name]
#    action: replace
#    target_label: kubernetes_name

##AL NOT USED TODAY pods should be scrap flag to put to true
# Example scrape config for probing services via the Blackbox Exporter.
#
# The relabeling allows the actual service scrape endpoint to be configured
# via the following annotations:
#
# * `prometheus.io/probe`: Only probe services that have a value of `true`
#- job_name: 'kubernetes-services'
#
#  metrics_path: /probe
#  params:
#    module: [http_2xx]
#
#  kubernetes_sd_configs:
#  - role: service
#
#  relabel_configs:
#  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
#    action: keep
#    regex: true
#  - source_labels: [__address__]
#    target_label: __param_target
#  - target_label: __address__
#    replacement: blackbox
#  - source_labels: [__param_target]
#    target_label: instance
#  - action: labelmap
#    regex: __meta_kubernetes_service_label_(.+)
#  - source_labels: [__meta_kubernetes_service_namespace]
#    target_label: kubernetes_namespace
#  - source_labels: [__meta_kubernetes_service_name]
#    target_label: kubernetes_name

##AL NOT USED TODAY pods should be scrap flag to put to true
# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
# * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
#- job_name: 'kubernetes-pods'
#
#  kubernetes_sd_configs:
#  - role: pod
#
#  relabel_configs:
#  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
#    action: keep
#    regex: true
#  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
#    action: replace
#    target_label: __metrics_path__
#    regex: (.+)
#  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
#    action: replace
#    regex: (.+):(?:\d+);(\d+)
#    replacement: ${1}:${2}
#    target_label: __address__
#  - action: labelmap
#    regex: __meta_kubernetes_pod_label_(.+)
#  - source_labels: [__meta_kubernetes_pod_namespace]
#    action: replace
#    target_label: kubernetes_namespace
#  - source_labels: [__meta_kubernetes_pod_name]
#    action: replace
#    target_label: kubernetes_pod_name

# Monitoring of nodes via node-exporter
- job_name: 'kubernetes-node-exporter'
  scrape_interval: 60s
  scrape_timeout: 60s
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - role: node
  relabel_configs:
  #AL Scrape only the nodes labelled as "app", so dont scrape master/infra
  - source_labels: [__meta_kubernetes_node_label_role]
    regex: app.*
    action: keep

  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - source_labels: [__meta_kubernetes_role]
    action: replace
    target_label: kubernetes_role
  - source_labels: [__address__]
    regex: '(.*):10250'
    replacement: '${1}:9100'
    target_label: __address__
  - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]
    target_label: __instance__
  # set "name" value to "job"
  - source_labels: [job]
    regex: 'kubernetes-(.*)'
    replacement: '${1}'
    target_label: name

- job_name: 'zzz-jmx-exporter'
  # can take up to 40 s to retrieve JMX Joram queues
  scrape_interval: 60s
  scrape_timeout: 60s
  kubernetes_sd_configs:
  - role: endpoints
  # Scrape all pods __meta_kubernetes_endpoint_port_name="joram-5556-tcp" and __meta_kubernetes_service_name="*joram-pm*" and in own namespace
  # then we relabel the instance to have the service name
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoint_port_name]
    regex: prod-eaa;jmx-exporter.*
    action: keep
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod
  - source_labels: [__meta_kubernetes_pod_container_name]
    target_label: container
  # WA only for snmp collector which is in the SAME container as ip coll-ma but on port 5558
  # - source_labels: [__meta_kubernetes_pod_container_port_number,__meta_kubernetes_pod_container_name]
  #   regex: 5556;eaa-proptima-ip-collector-ma
  #   replacement: 'eaa-proptima-snmp'
  #   target_label: container
  # end WA
  # drop all verbose GC logs polutting the DB from joram and mdds but keep java_lang_GarbageCollector_LastGcInfo_memoryUsageAfterGc_used (?i)java_lang_GarbageCollector_LastGcInfo_memoryUsageAfterGc_used(?-i).*
  metric_relabel_configs:
  # - source_labels: [__name__]
  #   regex: 'java_lang_GarbageCollector_LastGcInfo_memoryUsageAfterGc_used'
  #   replacement: 'jvm_memory_bytes_used_after_gc'
  #   target_label: __name__
  - source_labels: [__name__]
    regex: 'java_lang_(.*)'
    action: drop
  # drop all joram garbage except queues
  - source_labels: [__name__]
    regex: 'Joram_0_Destination_PendingMessageCount'
    replacement: 'joram_pending_queue'
    target_label: __name__
  - source_labels: [__name__]
    regex: 'Joram_0_Destination_NbMsgsReceiveSinceCreation'
    replacement: 'joram_messages_received'
    target_label: __name__
  - source_labels: [__name__]
    regex: 'Joram_.*|AgentServer_.*'
    action: drop

- job_name: 'blackbox-exporter'
  # maybe to change the frequency of the scrape H3G 2 min
  scrape_interval: 60s
  scrape_timeout: 60s
  metrics_path: /probe
  params:
    module: [http_2xx]  # Look for a HTTP 200 response.
  kubernetes_sd_configs:
  - role: endpoints
  # Scrape all pods __meta_kubernetes_pod_container_port_number="8080" and __meta_kubernetes_endpoints_name="eaa-.*" and in own namespace
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoint_port_name, __meta_kubernetes_endpoints_name]
    regex: prod-eaa;8080-tcp;eaa-(.*)
    action: keep
  - source_labels: [__address__]
    target_label: __param_target
  - source_labels: [__param_target]
    target_label: instance
  - source_labels: [__meta_kubernetes_endpoints_name]
    target_label: endpoint
  - target_label: __address__
    replacement: eaa-platform-blackbox-exporter:9115

- job_name: 'ceph-exporter'
  # can take up to 45 s to retrieve ceph data at Airtel with 4 monitor nodes
  scrape_interval: 60s
  scrape_timeout: 60s
  kubernetes_sd_configs:
  - role: endpoints
  # Scrape all only pods with endpoint is ceph
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoints_name]
    regex: prod-eaa;eaa-platform-ceph-exporter
    action: keep

#replace the sql exporter+oracle agent
- job_name: 'db-exporter'
  # maximum for a Prometheus TSDB, this is because queries can be really long to process
  scrape_interval: 300s
  scrape_timeout: 300s
  kubernetes_sd_configs:
  - role: endpoints
  # Scrape all pods __meta_kubernetes_endpoint_port_name="db-exporter" and __meta_kubernetes_service_name="*joram-pm*" and in own namespace
  # then we relabel the instance to have the service name
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoint_port_name]
    regex: prod-eaa;db-exporter-(.*)
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    regex: db-exporter-(.*)
    replacement: '${1}'
    target_label: db_instance

- job_name: 'logstash-exporter'
  scrape_interval: 60s
  scrape_timeout: 60s
  kubernetes_sd_configs:
  - role: endpoints
  # Scrape all svc __meta_kubernetes_endpoint_port_name="joram-5556-tcp" and __meta_kubernetes_service_name="*joram-pm*" and in own namespace
  # then we relabel the instance to have the service name
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_endpoints_name, __meta_kubernetes_endpoint_port_name]
    regex: prod-eaa;eaa-platform-logstash;port-monitoring
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    target_label: instance

########################### STATIC ###########################
- job_name: 'kube-state-metrics-exporter'
  # can take up to 40 s to retrieve data at RnD
  scrape_interval: 60s
  scrape_timeout: 60s
  static_configs:
    - targets: ['eaa-platform-kubestate-exporter:8888']

- job_name: 'elasticsearch-exporter'
  scrape_interval: 60s
  scrape_timeout: 60s
  static_configs:
    - targets: ['eaa-platform-elasticsearch-exporter:9108']

# grafana metrics
- job_name: 'grafana-exporter'
  scrape_interval: 60s
  scrape_timeout: 60s
  static_configs:
    -  targets: ['eaa-platform-grafana:3000']

# Prometheus self-monitoring
- job_name: 'prometheus'
  static_configs:
    - targets: ['eaa-platform-prometheus:9090']
/etc/prometheus #

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

[prom-crash-26-02-19.log](https://github.com/prometheus/prometheus/files/2905977/prom-crash-26-02-19.log)

....

[simonpasquier]

Hmm, fatal error: sweep increased allocation count could be triggered by a data race in the code or some issue with the Go runtime/compiler. To clear out the data race cause, you would need to build the binary with the -race option. Would you be able to do that?

[mtoulon]

Hi Simon Thanks for the update I am checking if we can recompile as suggested, I will get back to you ASAP Regards Mike. From: Simon Pasquier <notifications@github.com> Sent: Wednesday, February 27, 2019 11:01 AM To: prometheus/prometheus <prometheus@noreply.github.com> Cc: Michael Toulon <Michael.Toulon@mycom-osi.com>; Author <author@noreply.github.com> Subject: Re: [prometheus/prometheus] Prometheus abnormal crash (#5273) Hmm, fatal error: sweep increased allocation count could be triggered by a data race in the code or some issue with the Go runtime/compiler. To clear out the data race cause, you would need to build the binary with the -race option. Would you be able to do that? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#5273 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUJSbg2-LO3BsTuBGpe207JYH0PxKSkZks5vRmVwgaJpZM4bSUBO>.

…

________________________________ This electronic message contains information from MYCOM OSI which may be privileged or confidential. The information is intended to be for the use of the individual(s) or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or any other use of the contents of this information is prohibited. If you have received this electronic message in error, please notify us by post or telephone (to the numbers or correspondence address above) or by email (at the email address above) immediately.