Invalid UTF-8 label -- Prometheus Error with VTS Exporter.

[jakirpatel]

Prometheus Version : v2.4.3
Error message :
error while linting: text format parsing error in line 501: invalid label value "\xfftest-api"

Problem information:

We are having 10 nginx servers with VTS module enabled. Somehow the client is making the wrong http request and nginx is throwing 502 error. In this scenario VTS is capturing the invalid character as well.

Prometheus is collecting the metrics from all nginx servers. So if someone hit the nginx with invalid http host then prometheus is scraping all the metrics even with invalid character. This is causing potential failure at prometheus layer.

Its throwing the error:
error while linting: text format parsing error in line 501: invalid label value "\xfftest-api"
We do have three different layers here Nginx , VTS module and Prometheus. Because of this we are potentially loosing our all metrics.

Is there any solution to have on these? Anyway we can tell the prometheus to remove the metric with invalid label and start functioning as usual . I think prometheus somehow should ignore these kind of metric to be logged in prometheus to avoid the failure of prometheus itself.

[jakirpatel]

@vozlt Anything we can do for the error that we have got with the VTS..

[vozlt]

@jakirpatel
Thanks the bug report.
I merged the pull/151.

[axingblog]

use release v0.2.1 ,have same problem

request uri : /ip/%E6%B1%89%E5%AD%97

[vozlt]

@axingblog
Thanks for reporting. I'll check it soon.

[topicgit]

@axingblog Thanks for reporting. I'll check it soon.

Has this issue been fixed? I'm facing the same problem with 'invalid UTF-8 label value'.
Thanks.

[u5surf]

@axingblog
Hi, I’m a new collaborator for this module.
Thanks for your all bug report!
First of all, sadly we could not reproduce on this behavior with this configuraion which I assumed your reproduced one. I’d like to reproduce this behavior, can you provide your reproduced one? It is possible to clarify what happen in escaping that.

nginx_vts_filter_bytes_total{filter="appname::nginx",filter_name="/ip/%E6%B1%89%E5%AD%97",direction="in"} 99
nginx_vts_filter_bytes_total{filter="appname::nginx",filter_name="/ip/%E6%B1%89%E5%AD%97",direction="out"} 314

nginx.conf

http {
    include       mime.types;
    default_type  application/octet-stream;

    vhost_traffic_status_zone;
    upstream web {
            zone backend 64k;
            server 127.0.0.1:9000;
    }
    server {
        listen       8000;
        location / {
            vhost_traffic_status_filter_by_set_key $request_uri appname::nginx;
            proxy_pass http://web;
        }
    }
    server {
        listen       80;
        gzip off;
        client_max_body_size 100M;
        location /status/ {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format prometheus;
        }
    }
}

@topicgit
Also, if you have any clues(e.g. you have a configuraion which can be reproduced assuredly.) please inform for us🙏

[u5surf]

ngx_http_vhost_status_escape_prometheus seems not to escape the character which is represented by byte sequences between 0x80 - 0x10ffff.

[u5surf]

@axingblog @topicgit @vozlt @SuperQ
I will fix it below.

Which is prefer to represent \\x encoding format or % one?

e.g. request

curl http://localhost:8000/ip/姹夊瓧

result

nginx_vts_filter_bytes_total{filter="appname::nginx",filter_name="/ip/%E5%A7%B9%E5%A4%8A%E7%93%A7",direction="in"} 90
nginx_vts_filter_bytes_total{filter="appname::nginx",filter_name="/ip/%E5%A7%B9%E5%A4%8A%E7%93%A7",direction="out"} 314