Enum scrape error on Prometheus 2.5.0

[atisu]

Bug Report

What did you do?

Wrote an exporter using Python prometheus-client 0.4.2. My metrics are exported as Enums.

What did you expect to see?

Prometheus should be able to scrape the data. It works fine with Prometheus 2.4.3, 2.3.2, but fails on Prometheus 2.5.0. (If I export the data as Gauge it works as expected.)

What did you see instead? Under which circumstances?

On the /targets page the error is displayed:

invalid metric type "stateset"

Environment

The exporter produces this data (cut of at [...] as only additional similar metrics follow) when set to Enum (Prometheus 2.5.0 fails to scrape it):

# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="2",minor="7",patchlevel="15",version="2.7.15"} 1.0
# HELP ambari_alert Ambari Alert - {'UNKNOWN': 0, 'BAD': -1, 'CRITICAL': 1, 'OK': 3, 'WARNING': 2}
# TYPE ambari_alert gauge
ambari_alert{ambari_alert="BAD",name="namenode_last_checkpoint"} 0.0
ambari_alert{ambari_alert="CRITICAL",name="namenode_last_checkpoint"} 0.0
ambari_alert{ambari_alert="WARNING",name="namenode_last_checkpoint"} 0.0
ambari_alert{ambari_alert="UNKNOWN",name="namenode_last_checkpoint"} 0.0
ambari_alert{ambari_alert="OK",name="namenode_last_checkpoint"} 1.0
ambari_alert{ambari_alert="BAD",name="hbase_regionserver_process_percent"} 0.0
ambari_alert{ambari_alert="CRITICAL",name="hbase_regionserver_process_percent"} 0.0
ambari_alert{ambari_alert="WARNING",name="hbase_regionserver_process_percent"} 0.0
ambari_alert{ambari_alert="UNKNOWN",name="hbase_regionserver_process_percent"} 0.0
ambari_alert{ambari_alert="OK",name="hbase_regionserver_process_percent"} 1.0
ambari_alert{ambari_alert="BAD",name="ams_metrics_collector_hbase_master_process"} 0.0
ambari_alert{ambari_alert="CRITICAL",name="ams_metrics_collector_hbase_master_process"} 0.0
ambari_alert{ambari_alert="WARNING",name="ams_metrics_collector_hbase_master_process"} 0.0
ambari_alert{ambari_alert="UNKNOWN",name="ams_metrics_collector_hbase_master_process"} 0.0
[…]

When I export the data as Gauge, all versions are able to scrape the data:

 HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="2",minor="7",patchlevel="15",version="2.7.15"} 1.0
# HELP ambari_alert Ambari Alert - {'UNKNOWN': 0, 'BAD': -1, 'CRITICAL': 1, 'OK': 3, 'WARNING': 2}
# TYPE ambari_alert gauge
ambari_alert{name="namenode_last_checkpoint"} 3.0
ambari_alert{name="hbase_regionserver_process_percent"} 3.0
ambari_alert{name="ams_metrics_collector_hbase_master_process"} 3.0
ambari_alert{name="datanode_heap_usage"} 3.0
ambari_alert{name="hbase_regionserver_process"} 3.0
ambari_alert{name="hbase_master_process"} 3.0
[...]

promtool check metrics does not produce any errors when checking both on all versions (2.5.0, 2.4.3, 2.3.2).

• System information:

  Tested on Linux: Linux 3.10.0-862.14.4.el7.x86_64 x86_64 / CentOS Linux release 7.6.1810 (Core)
  Tested on MacOS: Darwin 17.7.0 x86_64

• Prometheus version:

  • Prometheus 2.3.2 Linux/AMD64 -- works

    prometheus, version 2.3.2 (branch: HEAD, revision:
    71af5e2)
    build user: root@5258e0bd9cc1
    build date: 20180712-14:02:52
    go version: go1.10.3

  • Prometheus 2.4.3 MacOS/AMD64 -- works

    prometheus, version 2.4.3 (branch: HEAD, revision: 167a4b4)
    build user: root@1e42b46043e9
    build date: 20181004-08:46:10
    go version: go1.11.1

  • Prometheus 2.5.0 Linux/AMD64 -- does not work

    prometheus, version 2.5.0 (branch: HEAD, revision:
    67dc912)
    build user: root@578ab108d0b9
    build date: 20181106-11:40:44
    go version: go1.11.1

  • Prometheus 2.5.0 MacOS/AMD64 -- does not work

    prometheus, version 2.5.0 (branch: HEAD, revision: 67dc912)
    build user: root@578ab108d0b9
    build date: 20181106-11:45:24
    go version: go1.11.1

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090', 'localhost:2222']

• Logs:

level=warn ts=2018-12-04T12:25:23.247929Z caller=scrape.go:836 component="scrape manager" scrape_pool=prometheus target=http://localhost:2222/metrics msg="append failed" err="invalid metric type \"stateset\""

[brian-brazil]

Can you try with current master? This should already be fixed.

[atisu]

Yes, it is working with master. Thanks!