context deadline exceeded

[MarilynZ]

What did you do?

Tried to run Prometheus server in a ubuntu server and scrape metrics from node_exporter instance running in a few private ip server . (see environment section.)

Used a basic config which scrapes these private ip server, via its private ip address, every 60 seconds.

Noticed that this very basic target was showing as DOWN in the Prometheus dashboard targets view, with the error: "context deadline exceeded"

What did you expect to see?

Both target groups reporting as UP and scrapes completing quickly

What did you see instead? Under which circumstances?

DOWN "context deadline exceeded"

Sometimes, it will become a down state. a few hours later, It will become the up state , but I'm not doing anything..

It looks like that the scrape request timed out. But when I set scrape_timeout: 120s,still appear "context, deadline exceeded" error

Environment

• System information:
  DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
Linux 4.4.0-47-generic x86_64

• Prometheus version:

  prometheus, version 1.7.1 (branch: master, revision: 3afb3fffa3a29c3de865e1172fb740442e9d0133) build user: root@0aa1b7fc430d build date: 20170612-11:44:05 go version: go1.8.3

• Prometheus configuration file:

global:
  scrape_interval:     30s
  evaluation_interval: 30s

  external_labels:
      monitor: 'Video-monitor'

rule_files:
   - "rules/alert.rules"

scrape_configs:
  - job_name: 'Video_prometheus'
    static_configs:
      - targets: ['192.168.1.13:9090']
        labels:
          alias: video-prometheus

  - job_name: 'Video_node_exporter'
    scrape_interval: 120s
    scrape_timeout: 120s
    static_configs:
      - targets: ['192.168.1.13:9100']
        labels:
          alias: video-prometheus
      - targets: ['192.168.1.2:9100']
        labels:
          alias: video-master
      - targets: ['192.168.1.3:9100']
        labels:
          alias: video-node01
      - targets: ['192.168.1.4:9100']
        labels:
          alias: video-node02
      - targets: ['192.168.1.5:9100']
        labels:
          alias: video-node03
      - targets: ['192.168.1.6:9100']
        labels:
          alias: video-node04
      - targets: ['192.168.1.7:9100']
        labels:
          alias: video-node05
      - targets: ['192.168.1.8:9100']
        labels:
          alias: video-node06
         … … … … 

• Logs:

time="2017-10-25T12:39:05+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:39:24+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 19.442712172s." source="persistence.go:665" 
time="2017-10-25T12:40:32+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=51609 memoryChunks=401057 source="storage.go:1867" urgencyScore=1 
time="2017-10-25T12:40:32+08:00" level=error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped." chunksToPersist=51457 memoryChunks=385939 source="storage.go:1007" urgencyScore=1 
time="2017-10-25T12:40:48+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=46416 memoryChunks=387607 source="storage.go:1857" urgencyScore=0.678 
time="2017-10-25T12:40:48+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:41:15+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=47004 memoryChunks=390279 source="storage.go:1867" urgencyScore=0.824 
time="2017-10-25T12:41:33+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 45.583003977s." source="persistence.go:665" 
time="2017-10-25T12:41:35+08:00" level=info msg="Storage does not need throttling anymore." chunksToPersist=47012 memoryChunks=390413 source="storage.go:1015" urgencyScore=0.824 
time="2017-10-25T12:41:50+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=45292 memoryChunks=391119 source="storage.go:1857" urgencyScore=0.643 
time="2017-10-25T12:42:19+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:42:45+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 26.396031303s." source="persistence.go:665" 
time="2017-10-25T12:45:19+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:45:54+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 34.336368215s." source="persistence.go:665" 
time="2017-10-25T12:48:23+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:48:47+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 24.752371907s." source="persistence.go:665" 
time="2017-10-25T12:52:05+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:52:26+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 21.347779175s." source="persistence.go:665" 
time="2017-10-25T12:55:19+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:55:47+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 27.398922414s." source="persistence.go:665" 
time="2017-10-25T12:59:01+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T12:59:19+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 17.44142578s." source="persistence.go:665" 
time="2017-10-25T13:02:00+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=42452 memoryChunks=382769 source="storage.go:1867" urgencyScore=0.894 
time="2017-10-25T13:02:01+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=41925 memoryChunks=382769 source="storage.go:1857" urgencyScore=0.685 
time="2017-10-25T13:02:01+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T13:02:30+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 29.087830198s." source="persistence.go:665" 
time="2017-10-25T13:02:37+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=41838 memoryChunks=383493 source="storage.go:1867" urgencyScore=0.844 
time="2017-10-25T13:02:40+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=40930 memoryChunks=383563 source="storage.go:1857" urgencyScore=0.641 
time="2017-10-25T13:03:09+08:00" level=info msg="Completed maintenance sweep through 56789 archived fingerprints in 6h31m42.655936538s." source="storage.go:1442" 
time="2017-10-25T13:03:32+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 
time="2017-10-25T13:04:00+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 28.79803733s." source="persistence.go:665" 
time="2017-10-25T13:06:53+08:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633" 

Thanks

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.