Prometheus scrapes data for custom exporter with multiple identical HELP and TYPE when promtool fails

[JoeyG1973]

What did you do?
Wrote a custom exporter for Spectrum Performance Center that puts out duplicate HELP and TYPE statements. The custom exporter is being fixed on our end but Prometheus scraped the data without issue. Eventually data stopped collecting even though Prometheus showed the scrape as just fine.

How the exporter is called:

http://localhost:8001/int/myrouter1

Output:

# HELP interface_bw Interface BW for Gi1
# TYPE interface_bw summary
interface_bw_in_bits{device="myrouter1",interface="Gi1",} 11
interface_bw_out_bits{device="myrouter1",interface="Gi1",} 16
# HELP interface_bw Interface BW for Gi2
# TYPE interface_bw summary
interface_bw_in_bits{device="myrouter1",interface="Gi2",} 1477
interface_bw_out_bits{device="myrouter1",interface="Gi2",} 125709
# HELP interface_bw Interface BW for Gi3
# TYPE interface_bw summary
interface_bw_in_bits{device="myrouter1",interface="Gi3",} 132831
interface_bw_out_bits{device="myrouter1",interface="Gi3",} 8135

Discovered the issue when promtool choked on the duplicate help and type.

What did you expect to see?

Prometheus should fail the scraper job and put something in the logs

What did you see instead? Under which circumstances?

Prometheus went along fat dumb and happy until at some point after a few days it decided to stop collecting data on myrouter1

Environment

• System information:

  Linux 3.10.0-693.5.2.el7.x86_64 x86_64

• Prometheus version:

  prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d8)
  build user: root@6e784304d3ff
  build date: 20180119-12:01:23
  go version: go1.9.2

• Alertmanager version:

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     300s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'myrouter1'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    metrics_path: /int/myrouter1

    static_configs:
      - targets: ['SPCexporter:8001']

    basic_auth:
      username: nunya
      password: bidness

  - job_name: 'myrouter2'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    metrics_path: /int/myrouter2

    static_configs:
      - targets: ['SPCexporter:8001']

    basic_auth:
      username: nunya
      password: bidness

  - job_name: 'aws-gis-transit-prod-dx'
    static_configs:
      - targets: ['cloudwatch-exporter:9100']

  - job_name: 'Grafana Stats'
    scrape_interval: 5s
    static_configs:
      - targets: [ 'mygrafana' ]

• Alertmanager configuration file:

n/a

• Logs:

nothing came out in the logs

[brian-brazil]

I don't think your issue is related to the duplicates, though you should fix that too.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[JoeyG1973]

So prometheus not throwing an error on an illegal exporter data output is not an issue?

[brian-brazil]

This is as expected for Prometheus 2.x, it's a tradeoff for performance.

[JoeyG1973]

Ahh ok cool

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.