Error on ingesting samples with different value but same timestamp errors flood

[onorua]

What did you do?
Upgraded to 1.5.2 and run it for some time.

Environment and logs
Dockerized Prometheus node with 256GB of RAM and 12cores, monitoring around 1.5k nodes, at least 3 endpoints on each host.

• System information:

# uname -srm
Linux 4.4.0-47-generic x86_64

• Prometheus version:

prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@1a01c5f68840
  build date:       20170210-16:23:28
  go version:       go1.7.5

• Alertmanager version:

alertmanager, version 0.5.1 (branch: master, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user:       root@fb407787b8bf
  build date:       20161125-08:14:40
  go version:       go1.7.3

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     30s # By default, scrape targets every 15 seconds.
  evaluation_interval: 30s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'bhss'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "recording.rules"
  - "alert.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  # Auto-discovery via consul tags by Registrator
  - job_name: 'consul-services'

    consul_sd_configs:
    - server: 'consul:8500'
      scheme: 'http'


    metric_relabel_configs:
      - source_labels: [__name__]
        regex: go_(.*)
        action: drop
      - source_labels: [__name__]
        regex: container_(fs|network|spec)_(.*)
        action: drop
      - source_labels: [__name__]
        regex: http_(.*)
        action: drop
      - source_labels: [__name__]
        regex: node_(net|sock|vm)stat_(.*)
        action: drop
      - source_labels: [__name__]
        regex: prometheus_(.*)
        action: drop
      - source_labels: [__name__]
        regex: docker_container_(blkio|net)(.*)
        action: drop
      - source_labels: [__name__]
        regex: net_(icmp|tcp|udp|ip)(.*)
        action: drop
      - source_labels: [__name__]
        regex: nstat_(.*)
        action: drop
      - target_label: id
        replacement: ''
      - target_label: container_label_com_docker_compose_config_hash
        replacement: ''
      - target_label: container_label_build_date
        replacement: ''
      - target_label: container_label_com_docker_compose_container_number
        replacement: ''
      - target_label: container_label_com_docker_compose_oneoff
        replacement: ''
      - target_label: container_label_com_docker_compose_project
        replacement: ''
      - target_label: container_label_com_docker_compose_service
        replacement: ''
      - target_label: container_label_com_docker_compose_version
        replacement: ''
      - target_label: container_label_license
        replacement: ''
      - target_label: container_label_name
        replacement: ''
      - target_label: io_kubernetes_container_name
        replacement: ''
      - target_label: io_kubernetes_pod_name
        replacement: ''
      - target_label: io_kubernetes_container_restartCount
        replacement: ''

    relabel_configs:
    - source_labels: [ '__meta_consul_tags' ]
      action: keep
      regex: .*exporter.*
    - source_labels: [ '__meta_consul_address', '__meta_consul_service_port' ]
      action: replace
      regex: (.+)(?::\d+);(\d+)
      replacement: $1:$2
      target_label: __address__
    - source_labels: [ '__meta_consul_service' ]
      action: keep
      regex: (.+)
      replacement: $1
      target_label: __name__
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: (.*):(.*):(.*)
      replacement: $2
      target_label: container_name
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: (.*):(.*):(.*)
      replacement: $1
      target_label: host_name
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: .*-(\w*)-(\w*)-(\d*):(.*):(.*)
      replacement: $1
      target_label: colo
    - source_labels: [ '__meta_consul_service_id' ]
      action: replace
      regex: (\w*-\w*-\w*)-(\d*):(.*):(.*)
      replacement: $1
      target_label: host_type

• Logs:

ime="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1136 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1135 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1123 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1163 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1226 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1124 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1163 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1146 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1117 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1179 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1212 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1160 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1075 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1155 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1317 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1141 source="scrape.go:520"
time="2017-03-12T20:17:54Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1128 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1102 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1154 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1127 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1145 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1123 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1368 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1144 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1176 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1136 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1223 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1152 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1051 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1219 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1116 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1063 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1082 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1100 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1185 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1111 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1185 source="scrape.go:520"
time="2017-03-12T20:17:55Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1233 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1229 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1069 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1082 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1071 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1138 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1299 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1083 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1246 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1132 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:17:56Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1206 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1147 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1161 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1116 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1150 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1079 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1283 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1138 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1150 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1072 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1177 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1127 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1166 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1232 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1099 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1136 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1128 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1147 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1200 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1374 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:17:57Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1144 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1186 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1139 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1155 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1162 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1178 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1115 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1152 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1161 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1189 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1076 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1176 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1186 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:17:58Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1163 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1080 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1142 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1132 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1148 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1074 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1363 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1223 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1216 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1196 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1175 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1100 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1134 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1148 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1099 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1115 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1169 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:17:59Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1159 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1077 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1200 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1111 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1132 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1192 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1082 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1176 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1115 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1126 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1068 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1332 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1100 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1344 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1146 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1162 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1348 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1181 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1185 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1229 source="scrape.go:520"
time="2017-03-12T20:18:00Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1020 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1073 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1064 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1173 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1111 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1148 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1177 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1146 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1145 source="scrape.go:520"
time="2017-03-12T20:18:01Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1079 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1088 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1130 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1133 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1092 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1193 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1107 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1196 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1212 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1172 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1207 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1129 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1135 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1072 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1140 source="scrape.go:520"
time="2017-03-12T20:18:02Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1190 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1092 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1083 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1151 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1067 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1116 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:18:03Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1231 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1124 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1142 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1059 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1139 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1083 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1128 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1061 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1133 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1189 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:612"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1149 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1192 source="scrape.go:520"
time="2017-03-12T20:18:04Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1177 source="scrape.go:520"
time="2017-03-12T20:18:05Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1074 source="scrape.go:520"
time="2017-03-12T20:18:21Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=787 source="scrape.go:520"
time="2017-03-12T20:18:22Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1201 source="scrape.go:520"
time="2017-03-12T20:18:23Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:23Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1198 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1161 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1110 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1109 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1137 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1193 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1059 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1094 source="scrape.go:520"
time="2017-03-12T20:18:24Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1125 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1087 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1118 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1202 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1164 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1084 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1056 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1228 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1068 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1091 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1173 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1204 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1239 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1102 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=990 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1103 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1159 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1097 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1154 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1191 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1122 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1125 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1120 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1096 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1065 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1121 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1141 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1119 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1215 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1113 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1159 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1105 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1155 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1157 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1182 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1165 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1206 source="scrape.go:520"
time="2017-03-12T20:18:25Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1078 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1138 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1017 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1093 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1081 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1108 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1183 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1114 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1085 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1143 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1025 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1086 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1179 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1089 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1160 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1184 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1101 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1135 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1095 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1179 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1158 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1090 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1144 source="scrape.go:520"
time="2017-03-12T20:18:26Z" level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1206 source="scrape.go:520"

[juliusv]

This error happens when multiple samples come in for a given series (metric name + unique labelset) for the same timestamp (likely from the same target scrape), but with a different value.

This is most likely because you have a bunch of relabeling rules that just drop labels, like this:

      - target_label: id
        replacement: ''
      - target_label: container_label_com_docker_compose_config_hash
        replacement: ''
      - target_label: container_label_build_date
        replacement: ''
      - target_label: container_label_com_docker_compose_container_number
        replacement: ''
      - target_label: container_label_com_docker_compose_oneoff
        replacement: ''
      - target_label: container_label_com_docker_compose_project
        replacement: ''
      - target_label: container_label_com_docker_compose_service
        replacement: ''
      - target_label: container_label_com_docker_compose_version
        replacement: ''
      - target_label: container_label_license
        replacement: ''
      - target_label: container_label_name
        replacement: ''
      - target_label: io_kubernetes_container_name
        replacement: ''
      - target_label: io_kubernetes_pod_name
        replacement: ''
      - target_label: io_kubernetes_container_restartCount
        replacement: ''

Some of those labels are probably needed to uniquely identify the series that have them - and dropping them maps multiple series into the same labelset, which is invalid. You can't just drop labels that are needed for uniqueness.

[onorua]

I drop labels because I do not want to create new metrics each time I recreate a container. Is there any way to drop that labels not causing any troubles?

[brian-brazil]

For usage questions, https://groups.google.com/forum/#!forum/prometheus-users is your best bet.

[xixikaikai]

I've also the same question, but I do not know which metric causing the trouble, and prometheus log does not show this error, only the msg="Error on ingesting samples with different value but same timestamp"

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.