Invalid UTF-8 label - VTS Nginx Exporter

[jakirpatel]

Prometheus Version : v2.4.3
VTS module: https://github.com/vozlt/nginx-module-vts
Error message :

error while linting: text format parsing error in line 501: invalid label value "\xfftest-api"

Problem information:

We are having 10 nginx servers with VTS module enabled. Somehow the client is making the wrong http request and nginx is throwing 502 error. In this scenario VTS is capturing the invalid character as well.

Prometheus is collecting the metrics from all nginx servers. So if someone hit the nginx with invalid http host then prometheus is scraping all the metrics even with invalid character. This is causing potential failure at prometheus layer.

Its throwing the error:
error while linting: text format parsing error in line 501: invalid label value "\xfftest-api"

We do have three different layers here Nginx , VTS module and Prometheus. Because of this we are potentially loosing our all metrics.

Is there any solution to have on these? Anyway we can tell the prometheus to remove the metric with invalid label and start functioning as usual . I think prometheus somehow should ignore these kind of metric to be logged in prometheus to avoid the failure of prometheus itself.

[brian-brazil]

You'll need to take this up with the VTS Nginx exporter, as it is not producing valid text format. There's nothing we or you can do here on the Prometheus side.

[jakirpatel]

@brian-brazil This make sense that this problem must be solved at exporter level. But for quick workaround, can't we do anything at prometheus level ??

[brian-brazil]

No, there's nothing you can do in Prometheus. Invalid output will be rejected before it gets to anything you can control.

[jakirpatel]

Thanks @brian-brazil for your reply.. I ll check the exporter upstream...