context deadline exceeded

[lcbluestorm]

the prometheus.yml configuration is:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.

  external_labels:
    monitor: 'codelab-monitor'


rule_files:
  - "recoding.rules"
  - "alert.rules"


scrape_configs:
  - job_name: 'prometheus'

    scrape_interval: 5s
    scrape_timeout: 3s

    target_groups:
      - targets: ['localhost:19089']
        labels:
          group: 'self'
- job_name: 'article_service'

    scrape_interval: 5s
    scrape_timeout: 3s
    target_groups:
      - targets: ['localhost:9711']
        labels:
          group: 'articleService-test'

But, there is a error like that:

[fabxc]

This simply means that the scrape request timed out. Try setting your scrape timeout to something higher than 3s or figure out whether it will always time out for some other reason.

We can probably map the user-facing error to something clearer.

[lcbluestorm]

Does the prometheus scrape from target by http protocol?

[fabxc]

Yes

On Tue, Mar 1, 2016, 12:13 PM lcbluestorm notifications@github.com wrote:

Does the prometheus scrape from target by http protocol?

—
Reply to this email directly or view it on GitHub
#1438 (comment)
.

[lcbluestorm]

So, for the target application with http protocol, the prometheus can not scrape from it?

[fabxc]

The request is timing out. I cannot tell you the reason without further information.
If you click on the target link that you've shown us a picture above you might find out more.

You should see a text page listing metrics exposed by that target.

[autrejacoupa]

The curl was able to pull data less than 1 sec "time curl http://10.8.12.37:42000/metrics" . Increased METRICS_RESOLUTION=5s helped. Restart the docker for pmm-server with this option and not pmm-data

sudo docker run -d \ -p 80:80 \ --volumes-from pmm-data \ --name pmm-server \ --restart always \ -e METRICS_RESOLUTION=5s \ percona/pmm-server:1.0.3

[heschmidt04]

For help with debugging for their Kubernetes Prometheus target of Prometheus pushgateway (quay image v.0.3.1 and 0.3.0) for the context deadline exceeded error.

Several things came into play for my "context deadline exceeded issue" when working in Kubernetes cluster with Prometheus Pushgateway (not Prometheus itself --> the Targets page only bubbled up the issue as the target was DOWN and had context deadline exceeded for an error message)

0. Context: Using this job_name pattern for the scrape_config in Kubernetes for Prometheus Config Map

    - job_name: 'pushgateway-metrics'
      static_configs:
      - targets: ['pushgateway.prometheus.svc.cluster.local:9091']

1. The Kubernetes cluster Pushgateway Deployment YAML file, deployed in the prometheus namespace, needed an extra - in front of the startup args for the pushgateway executable as well as the - for being the member of the array.
   Example below:

     - args:
        - -web.telemetry-path=/metrics

2. Getting into the Kubernetes pushgateway pod itself, because it was busybox and a stripped down version of OS, meant that the command telnet appname.namespace 9091 will not return any code/error or state it had connected with a banner (so I assumed it had hung)

3. On the Kubernetes pushgateway pod itself the netstat -l -p showed the port 9091 was bound and listening for process pushgateway. Tests from other pods with telnet or curl hung.

4. The clincher was that the service Selector was name: and not app: --> Once that was fixed the target was discovered on the Kubernetes Dashboard because the Kubernetes service knew to look for application name pushgateway versus what would be the pod name eg: pushgateway-2976552050-nxrkb

Hope that helps someone looking for similar info.

[billykwooten]

To add to this for future bypassers. I configured pi-hole on my network to stop ads, by changing pi-hole to my main DNS, it affected my containers, even with conditional forwarding turned on.

I had to pass the --dns flag to my container and specify my main dns server to get it to quit flapping.