Error: text format parsing error in line 1: invalid metric name

[bsushant-athena]

My configuration:

global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
monitor: "gatewaymonitor"

scrape_configs:

• job_name: "gateway"
  metrics_path: "/public/metrics"

  static_configs:

  • targets: ["10.8.110.83:18888"]
    I'm using spring cloud zuul/hystrix/ribbon all the netflix oss.
    I'm able to see the hystrix metrics here>> http://10.8.110.83:18888/public/metrics

I'm not able to see any issue with my prometheus.yml.
The error I'm getting: https://ibb.co/hDXR5w
Please help.

[brian-brazil]

Are you using client_java?

[bsushant-athena]

I'm currently using this one >> https://github.com/ahus1/prometheus-hystrix

[brian-brazil]

That's a bug to raise with them if they're producing invalid metric names.

[checketts]

@bsushant-athena I've been investigating the hystrix prometheus client myself, can you post what the /metrics endpoint is producing? (As in what the invalid name likely is?)

I can't hit the endpoint you posted above myself

[bsushant-athena]

you can access the /metrics output here>> http://jsoneditoronline.org/?id=26dbc05242d6f12bba63df3f31ece107

[checketts]

That output is from the spring actuator metrics and is not in the Prometheus format, in fact that is json. You need to hit the actual Prometheus endpoint.

[bsushant-athena]

Thanks, Do you have any reference for this i.e. How to get the actual metrics?

[checketts]

If you are using https://github.com/prometheus/client_java/tree/master/simpleclient_spring_boot then the metrics would be exposed at /prometheus

Please use stack overflow for further help requests. Issues are intended for bugs

[bsushant-athena]

hey, I got the hystrix metrics >> http://jsoneditoronline.org/?id=2f71c34ed76155ddb92d94743793cbac but I'm not able to format them in prometheus format. Can you suggest a blog/something which describes way to publish hysteria stream metrics to prometheus ? I tries with prometheus-hystrix but that is not working !

[bsushant-athena]

today I integrated the spring_boot_client with the prometheus-hystrix library and changed the endpoint to /prometheus and now getting this data >> http://jsoneditoronline.org/?id=5d1a431c7cd9c72fa07d16eb075f4267
but the error is >> text format parsing error in line 109: second HELP line for metric name "counter_servo_eurekaclient_transport_request"
can you please tell me what is the issue here ?

[brian-brazil]

Exactly what it says, you have duplicate metrics. It looks like you're missing labels.

[bsushant-athena]

sorry ,could you please elaborate more on labels ?

[checketts]

@bsushant-athena These are very beginning questions. Please read the Prometheus site to learn about how it works: https://prometheus.io/docs/instrumenting/exposition_formats/

Also we've seen the servo metrics report incorrectly in the past. I don't have any experience with where those are coming from, so but it appears it is trying to take the raw Spring boot metrics and convert those to be exposed in Prometheus. Look into the Micrometer.io converter for Spring: https://github.com/micrometer-metrics/micrometer/tree/master/micrometer-spring-legacy since it migrates those metrics properly

[brian-brazil]

Note that the Servo metrics are known to be broken, as they don't follow the Spring API (unless they've fixed it recently).

[bsushant-athena]

OK I followed up some of the recommendations from older issues, and I'm able to see spring boot metrics in prometheus format one /prometheus endpoint. But still not able to get the hystrix metrics !