/metrics sometimes return "has a label named 'service' whose value is not utf8"

[salehjafarli]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Prometheus sometimes fails to scrap /metrics endpoint and gets 500. Sent request to /metrics endpoint from browser.

What did you see instead?

collected metric "traefik_router_request_duration_seconds" { label:<name:"code" value:"200" > label:<name:"method" value:"GET" > label:<name:"protocol" value:"http" > label:<name:"router" value:"nomad@file" > label:<name:"service" value:"nomad@file\000\000\000\000\000\000+\225\t\000\000\000\000\000" > histogram:<sample_count:1 sample_sum:0.038218447 bucket:<cumulative_count:1 upper_bound:0.1 > bucket:<cumulative_count:1 upper_bound:0.3 > bucket:<cumulative_count:1 upper_bound:1.2 > bucket:<cumulative_count:1 upper_bound:5 > > } has a label named "service" whose value is not utf8: "nomad@file\x00\x00\x00\x00\x00\x00+\x95\t\x00\x00\x00\x00\x00"

Sometimes when request is successful, among scraped metrics i can see these ones:

traefik_router_request_duration_seconds_count{code="200",method="GET",protocol="http",router="nomad@file",service="accountant@consulcatalog"} 185
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file",le="0.1"} 326450
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file",le="0.3"} 326559
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file",le="1.2"} 326680
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file",le="5"} 329438
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file",le="+Inf"} 330085
traefik_router_request_duration_seconds_sum{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file"} 126980.5491633237
traefik_router_request_duration_seconds_count{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file"} 330085
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@",le="0.1"} 1
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@",le="0.3"} 1
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@",le="1.2"} 1
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@",le="5"} 1
traefik_router_request_duration_seconds_bucket{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@",le="+Inf"} 1
traefik_router_request_duration_seconds_sum{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@"} 0.038218447
traefik_router_request_duration_seconds_count{code="200",method="GET",protocol="http",router="nomad@file",service="nomad@file�������������@"} 1

Could it be issue on traefik?

What version of Traefik are you using?

2.8.1

What is your environment & configuration?

Traefik v2.8.1 running on Nomad

api:
  insecure: true
  dashboard: true
ping: {}
entryPoints:
  websecure:
    address: ":443"
  traefik:
    address: ":21002"
  kafka:
    address: ":9094"   
  rabbitmq:
    address: ":5672"  
providers:
  consulcatalog:
    refreshInterval: 5s
    exposedByDefault: false
    endpoint:

      address: http://****:8500
      token: ****
    prefix: traefik
    connectAware: true
    connectByDefault: false
  file:
    filename: "local/traefik.yml"
tls:
  certificates:
    - certFile: local/certificate.pem
      keyFile: local/key.pem  
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        # tls 1.3
      - TLS_AES_256_GCM_SHA384
      - TLS_AES_128_GCM_SHA256
      - TLS_CHACHA20_POLY1305_SHA256    
    mintls13:
      minVersion: VersionTLS13
http:
  routers:
    nomad:
      entryPoints:
        - websecure
      service: nomad
      rule: "Host(`nomad.****.com`)"
      tls: true
    consul:
      entryPoints:
      service: consul
        - websecure
      rule: "Host(`consul.****.com`)"
      tls: true
    traefik:
      entryPoints:
        - websecure
      service: api@internal
      rule: "Host(`traefik.****.com`)"
      tls: true
    vault:
      entryPoints:
        - websecure
      service: vault
      rule: "Host(`vault.****.com`)"
      tls: true                              
  services:

    nomad:
      loadBalancer:
        servers:
          - url: http://****:4646
    consul:
      loadBalancer:
        servers:
          - url: http://****:8500
    vault:
      loadBalancer:
        servers:
          - url: http://****:8200

log:
  level: INFO
accessLog: {}
metrics:
  prometheus:
    addRoutersLabels: true
    addServicesLabels: true

If applicable, please paste the log output in DEBUG level

No response

[jspdown]

Hello @salehjafarli 👋

Could you try with the latest v2.11 version?
It should be fixed by #9241

[salehjafarli]

Hi @jspdown, yes, it is fixed after upgrade to 2.11.2, closing issue now. But I was wondering, could you briefly tell me what the problem causing this issue was? Thanks.

[m198799]

Traefik 2.6.7 had the same problem
collected metric "traefik_router_open_connections" { label:<name:"method" value:"POST" > label:<name:"protocol" value:"http" > label:<name:"router" value:"agentsecure-qt-system-ir-ids-endpoint-client-8a4267664ad3d1062eb0@kubernetescrd\000\314" > label:<name:"service" value:"qt-system-ir-service-agent2-client-b7489e5c775768b00473@kubernetescrd" > gauge:<value:0 > } has a label named "router" whose value is not utf8: "agentsecure-qt-system-ir-ids-endpoint-client-8a4267664ad3d1062eb0@kubernetescrd\x00\xcc"
I tried upgrading to see if it could be fixed