Error on ingesting out-of-order samples

[vincent927]

What did you do?
I started a node_exporter , and started a prometheus server on localhost

What did you expect to see?

What did you see instead? Under which circumstances?

WARN[0035] Error on ingesting out-of-order samples       numDropped=520 source=scrape.go:533
WARN[0035] Scrape health sample discarded                error=sample timestamp out of order sample=up{alias="node1", instance="node1", job="linux"} => 1 @[1495195178.231] source=scrape.go:586
WARN[0035] Scrape duration sample discarded              error=sample timestamp out of order sample=scrape_duration_seconds{alias="node1", instance="node1", job="linux"} => 0.048101537 @[1495195178.231] source=scrape.go:589
WARN[0035] Scrape sample count sample discarded          error=sample timestamp out of order sample=scrape_duration_seconds{alias="node1", instance="node1", job="linux"} => 0.048101537 @[1495195178.231] source=scrape.go:592
WARN[0035] Scrape sample count post-relabeling sample discarded  error=sample timestamp out of order sample=scrape_duration_seconds{alias="node1", instance="node1", job="linux"} => 0.048101537 @[1495195178.231] source=scrape.go:595

Environment
Centos6.5

• System information:

Linux 2.6.32-431.el6.x86_64 x86_64

• Prometheus version:

prometheus, version 1.6.2 (branch: master, revision: b38e977)
build user: root@c99d9d650cf4
build date: 20170511-12:59:13
go version: go1.8.1

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
#  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

#    static_configs:
#      - targets: ['localhost:9090']
  - job_name: 'linux'
    static_configs:
        - targets: ['192.168.130.1:9100']
          labels:
            instance: node1
            alias: node1
#  - job_name: 'mysql'
#    static_configs:
#        - targets: ['192.168.130.1:9100']
#          labels:
#            instance: node1```

* Logs:

WARN[0035] Error on ingesting out-of-order samples numDropped=520 source=scrape.go:533
WARN[0035] Scrape health sample discarded error=sample timestamp out of order sample=up{alias="node1", instance="node1", job="linux"} => 1 @[1495195178.231] source=scrape.go:586
WARN[0035] Scrape duration sample discarded error=sample timestamp out of order sample=scrape_duration_seconds{alias="node1", instance="node1", job="linux"} => 0.048101537 @[1495195178.231] source=scrape.go:589
WARN[0035] Scrape sample count sample discarded error=sample timestamp out of order sample=scrape_duration_seconds{alias="node1", instance="node1", job="linux"} => 0.048101537 @[1495195178.231] source=scrape.go:592
WARN[0035] Scrape sample count post-relabeling sample discarded error=sample timestamp out of order sample=scrape_duration_seconds{alias="node1", instance="node1", job="linux"} => 0.048101537 @[1495195178.231] source=scrape.go:595

[brian-brazil]

Time on your machine is likely broken, please check your NTPd.

[vincent927]

I have been synchronized with NTP, but still being given

ntpdate asia.pool.ntp.org

[brian-brazil]

If time jumped backwards, wait for it to catch back up to where it was.

[vincent927]

I still do not understand how I can solve this problem。
@brian-brazil

[gouthamve]

Hi @vincent927 This is caused due to the following issue.

Suppose, for example, your current unix time is 1500 and due to a time sync the time went back to 1400 any new samples that are ingested with time less than 1500 will be deemed out of order. You need to wait for your system time to catch up to 1500 again.

[vincent927]

@gouthamve
Thank you so much .
But is it only waiting for this one?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.