Metrics not collected when there are bogus characters on /metrics page

[ajardan]

What did you do?
Collecting metrics from traefik proxy

What did you expect to see?
metrics to be saved in prometheus

What did you see instead? Under which circumstances?
no metrics were collected when there were bogus charactes in /metrics page

(see containous/traefik#1821)
Environment

• System information:

Linux 3.10.0-514.21.1.el7.x86_64 x86_64

• Prometheus version:

  prometheus, version 1.6.1 (branch: master, revision: 4666df5)
  build user: root@7e45fa0366a7
  build date: 20170419-14:32:22
  go version: go1.8.1

• Prometheus configuration file:

global:
  scrape_interval:     30s
  evaluation_interval: 30s

  labels:
    cluster: swarm
    replica: "1"

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'prometheus-swarm'

rule_files:
  - "alert.rules_nodes"
  - "alert.rules_tasks"
  - "alert.rules_service-groups"
  - job_name: 'load-balancer'
    dns_sd_configs:
    - names:
      - 'tasks.load-balancer'
      type: 'A'
      port: 9080```

* Logs:

unfortunately, there's nothing in the logs```

[brian-brazil]

What do you mean by bogus characters?

The intended behaviour is that data that does not meet the requirements of the exposition format will lead to a failed scrape.

[gouthamve]

Yes, I don't think the label in the linked issue is valid:

method="Ý̪î����rgda����������������������������������������ù������������������������������������������������������������������������������������������������������������������������������������������PY£Ÿx������������������������������������������������������������������������������������������������������������������������com.apple.WebKit����GET"

We only support UTF-8. See: https://prometheus.io/docs/instrumenting/exposition_formats/

[ajardan]

Thank you @brian-brazil and @gouthamve

I completely agree this is not valid, but the issue is: nothing is collected when such an entry is in the /metrics page.

I opened a ticket with traefik, and they say it is a bogus method sent by a client, so it is displayed as is in the metrics page. Recommended to open a ticket with Prometheus, which seems like a valid point, since collection should not be aborted completely when just a few labels are wrong ?

[brian-brazil]

The fault here is presumably with an invalid utf-8 string being provided to the client, this needs fixing on traefik's end.

since collection should not be aborted completely when just a few labels are wrong

Arbitrary partial scrapes would be nigh impossible to deal with correctly by users, and cause subtle failures. A hard failure is better.

[ajardan]

@brian-brazil thanks. Will try to convince traefik guys that it has to be fixed on their end, since they said it is a prometheus issue :)

But yeah, it looks like a format violation, so hopefully I will manage to convince them, otherwise the 2 tools (which basically cannot be used separately in the modern world) would not be useable together...

[ldez]

You don't need to convince me 😉

[ajardan]

@ldez thanks a lot ! :)

[ajardan]

I guess this can be closed.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.