Error : sample timestamp out of order

[txdyzdy]

What did you do?
Changed the server time, backwards for 12 hours and then immediately recovered.

What did you see instead? Under which circumstances?
Prometheus reported "Sample discarded" and "sample timestamp out of order" error.

I was told that the server uses its own time to ingest, so everything has samples from the future.once the server time has caught up, everything is back to normal.
AAfter a day Prometheus is now back to normal.
I want to know more about it. What is the mechanism?How is the system going to catch up?

your time probably jumped backwards when you enabled NTP. Prometheus cannot ingest samples from the past
if your time shift is not overly long, you can just wait until time has caught up to your previous notion of the present

Environment

• System information:

Linux 2.6.32-504.el6.x86_64 x86_64

• Prometheus version:

prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@a8af9200f95d
  build date:       20170210-14:41:22
  go version:       go1.7.5

• Logs:

time="2017-03-14T04:27:42-04:00" level=debug msg="Sample discarded" error="sample timestamp out of order" sample=node_cpu{cpu="cpu15", instance="10.10.20.66:9122", job="Host", mode="softirq"} => 0 @[1489480060.918] source="scrape.go:507" 

[brian-brazil]

Usage questions are best asked at https://groups.google.com/forum/#!forum/prometheus-users

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.