Occasional "Error on ingesting out-of-order samples" warnings in the log

[hasso]

There are occasional warnings in my prometheus log:

Jun 29 08:38:40 collector prometheus[13853]: time="2017-06-29T08:38:39Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=53 source="scrape.go:534"

Switching log level to debug reveals that it's complaining about internal data of collectd_exporter:

Jun 29 08:38:39 collector prometheus[13853]: time="2017-06-29T08:38:39Z" level=debug msg="Sample discarded" error="sample timestamp out of order" sample=go_gc_duration_seconds{job="collectd", quantile="0"} => 0.000074903 @[1498725519.678] source="scrape.go:524"
Jun 29 08:38:39 collector prometheus[13853]: time="2017-06-29T08:38:39Z" level=debug msg="Sample discarded" error="sample timestamp out of order" sample=go_gc_duration_seconds{job="collectd", quantile="0.25"} => 0.00012248 @[1498725519.678] source="scrape.go:524"
etc

It's always a single collectd_exporter – we are running multiple ones with 10s scrape interval and there is always interval multiple of 10s between warnings. And it's always collectd_exporter internal data only, never collectd collected data.

Environment

collector ~ $ prometheus -version
prometheus, version 1.7.1 (branch: master, revision: 3afb3fffa3a29c3de865e1172fb740442e9d0133)
  build user:       root@0aa1b7fc430d
  build date:       20170612-11:44:05
  go version:       go1.8.3
collector ~ $ collectd_exporter -version
collectd_exporter, version 0.3.1 (branch: master, revision: f8625a17c7713661de1a593a28875531f0873afa)
  build user:       root@builder
  build date:       20170612-09:45:53
  go version:       go1.8.3
collector ~ $ uname -srm
Linux 4.9.34 x86_64
collector ~ $

• Prometheus configuration file:

global:
  scrape_interval:     10s
  evaluation_interval: 10s

scrape_configs:
  - job_name: 'collectd'
    static_configs:
      - targets:
        - 172.16.63.254:54001
        - 172.16.67.254:54001
        - 172.16.71.254:54001
        - 172.16.75.254:54001
        - 172.16.79.254:54001
        - 172.16.83.254:54001
        - 172.16.87.254:54001
        - 172.16.91.254:54001
    metric_relabel_configs:
      - source_labels: [exported_instance]
        regex: (.*)
        target_label: instance
        replacement: ${1}
      - source_labels: []
        target_label: exported_instance
        replacement: ""

[brian-brazil]

Usage questions are best asked on https://groups.google.com/forum/#!aboutgroup/prometheus-users

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.