Error on ingesting out-of-order samples

[altmannmarcelo]

Hi guys, I'm running prometheus with node_exporter and mysqld_exporter on a vm. It's a centos7 machine and I've prometheus and the exporters installed as a systemd service. Everytime I reboot the machine, prometheus starts and reports an error:

Aug 02 14:06:18 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:18-03:00" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{alias="db1", instance="127.0.0.1:9100...rce="scrape.go:503"
Aug 02 14:06:19 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:19-03:00" level=warning msg="Error on ingesting out-of-order samples" numDropped=57 source="scrape.go:467"
Aug 02 14:06:19 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:19-03:00" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{alias="db1", instance="127.0.0.1:9104", job="mysql"} => 1 @[...rce="scrape.go:500"
Aug 02 14:06:19 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:19-03:00" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{alias="db1", instance="127.0.0.1:9104...rce="scrape.go:503"
Aug 02 14:06:23 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:23-03:00" level=warning msg="Error on ingesting out-of-order samples" numDropped=621 source="scrape.go:467"
Aug 02 14:06:23 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:23-03:00" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{alias="db1", instance="127.0.0.1:9100", job="linux"} => 1 @[...rce="scrape.go:500"
Aug 02 14:06:23 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:23-03:00" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{alias="db1", instance="127.0.0.1:9100...rce="scrape.go:503"
Aug 02 14:06:24 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:24-03:00" level=warning msg="Error on ingesting out-of-order samples" numDropped=57 source="scrape.go:467"
Aug 02 14:06:24 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:24-03:00" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{alias="db1", instance="127.0.0.1:9104", job="mysql"} => 1 @[...rce="scrape.go:500"
Aug 02 14:06:24 blog.marceloaltmann.com prometheus[2885]: time="2016-08-02T14:06:24-03:00" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{alias="db1", instance="127.0.0.1:9104...rce="scrape.go:503"

I'm running prometheus, version 1.0.1 (branch: master, revision: be40190).

Any help is appreciated.

[beorn7]

It looks like you have managed to set up multiple targets that feed into the same time series.

[JeromeVi]

Hi, I'm facing this problem and I can't find what is this time stamp ?
T06:34:07Z in GMT

Could you help me ?

[tangjiaxing669]

@JeromeVi
I have encountered, how is the solution?

[beorn7]

It makes more sense to ask questions like this on the [prometheus-users mailing list](https://groups.google.com/forum/#!forum/pro\
metheus-users") rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your ques
tion, and the whole community can benefit from the answers provided.

[JeromeVi]

All: sorry for the spam maybe @beorn7 is right and it's not the right place to ask/answer questions anyway @tangjiaxing669 I haven't found a solution, I just worked with the timezone that was given

[tangjiaxing669]

OK.

[tangjiaxing669]

https://groups.google.com/forum/#!topic/prometheus-users/KXbbIgbBhHo

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.