Prometheus doesn't appear to actually scrape pushgateway

[smd1000]

Bug Report

What did you do?
We have a set of ephemeral batch jobs that push their metrics to a pushgateway (v0.5.2). The jobs successfully push to the pushgateway and we can see them on the pushgateway gui and when we scrape the endpoint manually (localhost:9091/metrics).

What did you expect to see?
We expected prometheus (v2.4.0) to scrape the pushgateway.

What did you see instead? Under which circumstances?
Nothing.

Environment
Prometheus/alertmanager/pushgateway all on the same host.

• System information:

Linux 4.14.67-66.56.amzn1.x86_64 x86_64

• Prometheus version:

prometheus, version 2.4.0 (branch: HEAD, revision: 068eaa5dbfce6c08f3d05d3d3c0bfd96267cfed2)
  build user:       root@d84c15ea5e93
  build date:       20180911-10:46:37
  go version:       go1.10.3

• Pushgateway version:

pushgateway, version 0.5.2 (branch: HEAD, revision: 231071b3e0a57b58df717bba5f4e5d88cac95389)
  build user:       root@c95e387ff56a
  build date:       20180615-16:11:56
  go version:       go1.10.3

• Prometheus configuration file:

global:
  scrape_interval:     60s
  evaluation_interval: 60s

  external_labels:
      monitor: 'prometheus-monitor'

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
    scheme: http
    timeout: 10s

rule_files:
  - "rules/*"

scrape_configs:

  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'cloudwatch'
    static_configs:
      - targets: ['localhost:9273']

  - job_name: 'newrelic'
    scrape_timeout: 30s
    static_configs:
      - targets: ['localhost:9126']

  - job_name: 'pushgateway'
    honor_labels: true
    static_configs:
      - targets: ['localhost:9091']

• Logs:
  I see no relevant messages or errors in prometheus log with debug on regarding pushgateway or scraping.

[simonpasquier]

You can check the status of the target in the Targets page of the Prometheus UI. You can also check the up{job="pushgateway"} metric: 1 means that the scrape was successful, 0 otherwise.

[smd1000]

Hi, pushgateway is shown as up:

However, the following metric is not getting pulled into prometheus:

As you can see the metric was pushed yesterday and is still in the pushgateway.

I'm expecting to see ctan_test2_job in the prometheus ui but it is not present.

[simonpasquier]

ctan_test2_job is the name of the job in the PushGateway, not a metric name. From the second screenshot, it looks like there's no metric for ctan_test2_job (probably you deleted all metrics associated to the job). If you query {exported_job="ctan_test2_job"} in Prometheus, you'll get the same result as what the PushGateway UI displays.

I'm closing it for now. If you have further questions, please use our user mailing list, which you can also search.

[smd1000]

Does the Value field not represent the metric value? Sorry, I missed that part of the screenshot.

[simonpasquier]

push_time_seconds is the name of the metric.

[KyleFromOhio]

This "but report" was extremely helpful for me...

The following example worked for me...

EDIT: /opt/prometheus/prometheus.yml (must have correct indentation)

job_name: 'pushgateway'
honor_labels: true
static_configs:
- targets: ['localhost:9091']

Restart prometheus service to load new scrape config.
Run/start the pushgateway service.
For testing i ran manually: $ /pathto/binary/file/pushgateway --web.listen-address=":9091"

Then send a curl request from remote server to gateway (make sure firewall is correct first)...
$ echo 'test_metric 3.15' | curl --data-binary @- http://YOURGATEWAY:9091/metrics/job/pushgateway/instance/yourremoteserver01

Then check if scraping ok: http://YOURPROMETHEUS:9090/targets
Then look for your metric (should be the last one at bottom: http://YOURGATEWAY:9091/metrics
You could check locally like: $ curl -vs http://localhost:9091/metrics | tail