Getting error "expected label name, got \"BCLOSE\""

[wraithfive]

Bug Report

I have a spring application that exposes metrics via micrometer. After upgrading to Prometheus 2.7.1 (from a 2.6.x release) when scrapping that application I now get the error "expected label name, got "BCLOSE""

Environment

• System information:

  Linux 3.10.0-957.1.3.el7.x86_64 x86_64

• Prometheus version:

  prometheus, version 2.7.1 (branch: HEAD, revision: 62e591f)
  build user: root@f9f82868fc43
  build date: 20190131-11:16:59
  go version: go1.11.5

• Prometheus configuration file:

global:
  scrape_interval:     1m  # Set the scrape interval. Default is every 1 minute. Can be specified in seconds.
  evaluation_interval: 1m  # Evaluate rules every 1 minute. The default is every 1 minute. Can be specified in seconds.
  # scrape_timeout is set to the global default (10s).

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'client'
    metrics_path: /service/metrics
    static_configs:
      - targets: ["host.docker.internal:8080"]

• Logs:

level=warn ts=2019-02-08T21:33:50.763146989Z caller=scrape.go:835 component="scrape manager" scrape_pool=client target=http://host.docker.internal:8080/service/metrics msg="append failed" err="expected label name, got \"BCLOSE\""

[wraithfive]

Here is a sample of the metrics being scraped.
metrics.txt

[codesome]

Looking at this unit test here

prometheus/pkg/textparse/openmetricsparse_test.go

Lines 333 to 336 in 4cb6c20

	{
	input: "a{b=\"c\",} 1\n",
	err: "expected label name, got \"BCLOSE\"",
	},

I suspect it is causing because of ,} at the end of your metrics, i.e. the content type of the target is "application/openmetrics-text", but the metrics don't comply with it.

Can you let us know that is the content/media type of http://host.docker.internal:8080/service/metrics?

[wraithfive]

Content-Type is text/plain;charset=ISO-8859-1

I also suspected the ,} at the end of the metrics but that's coming from simple client and has been for some time. I found several bug reports about it on that project including a PR to make it no longer end labels with the extra comma but they were all closed as "prometheus spec allows it".

[brian-brazil]

That charset isn't right, it should be utf-8, but I don't see any non-ASCII characters.

[wraithfive]

I think my content type is wrong still actually. I just realized the content type is dictated by the requests accept header. I wasn't sending that when testing with postman to get the metrics so I get text/plain. (Not sure about the charset. I'll have to investigate that separately in my app.)

If I set the request to header to Accept: application/openmetrics-text then that is what I get for content type on the response. This make me thing prometheus may be sending that in the request headers. Trying to confirm that now.

[wraithfive]

That seems to be the issue. The "Accept" header in the scrape request from Prometheus is application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1.

Looking at the following in prometheus code:

prometheus/scrape/scrape.go

Lines 452 to 454 in 4cb6c20

	const acceptHeader = `application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1`

It seems that prometheus is always going to request application/openmetrics-text now so simple client will need to stop appending the comma or prometheus will need some sort of option for the scrape config to specify it we want prometheus metrics formate or open metrics format.

[brian-brazil]

The simpleclient only produces Prometheus textformat, and sets the Content-Type accordingly. This sounds like a bug in your http handling.

[wraithfive]

You could be right. This is an inherited Spring MVC project that hadn't been updated for years when I took over. I've been slowly bringing it to the modern world and it may be that some of the annotations need to be updated to produce correct Content-Types. The behavior of the current annotation seems to be always as I described, set the Content-Type based on the "Accept" header. But there may be another option. Let me do some more researching.

I still think it might be a good idea to enable explicitly specifying a metric format type in the scrape config.

[brian-brazil]

We're sending appropriate headers, I don't think we should be adding configuration options for broken http servers.

[wraithfive]

Are you really sending appropriate headers though if you specify you only accept "openmetrics-text" but you also accept "text/plain" (or whatever the Content-Type of prometheus format metrics is)? Seems you should send the accept header for the context type expected.

Even stipulating that my http handling is not correct it seems it would also be incorrect for simple client to receive a request for open metrics and then produce prometheus metrics.

I may be thinking incorrectly here but from my point of view you'd be adding configuration options for correctly configured http servers and it's only broken ones that would be functioning based on your request headers.

[brian-brazil]

We accept both and that's what our Accept header says. The problem is that your response has mismatched content and Content-Type.

[wraithfive]

Ah yes! Sorry, I was missing that you specified both types in the Accept header!

I have tested using the exact accept header you supply and have managed to change my app to supply the correct Content-Type. Prometheus is scraping it now.