Support prometheus metrics with nginx-module-vts

[sysulq]

Recently I came across prometheus, and I tried to implement prometheus metrics with nginx-module-vts.

Here is nginx-vts-exporter, which is heavily used in our system.

Hope this can be helpful, and any questions would be pleasure 😆

[vozlt]

Thank you for publishing your code. 👍
I would like to add your nginx-vts-exporter link in nginx-module-vts's README.md if you allow.

[sysulq]

sure 😄

[discordianfish]

Lemme hijack this issue.. @vozlt With prometheus become the defactor standard of metrics exposition, would you be open to add prometheus metrics natively? It's a simple text based format and it wouldn't require users to somehow 'translate' it by using either @hnlq715 exporter or Kubernetes ingress controller which does the same: https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/customization/custom-vts-metrics-prometheus/README.md

[SuperQ]

This appears to be a dupe of #58.

[sysulq]

Hope this could help you guys, simple lua code generates nginx's prometheus metrics.
https://github.com/hnlq715/nginx-prometheus-metrics

[ghost]

#114

[vozlt]

That feature has added.
All mertric names are started with nginx_vts_(*).
I can not guarantee that it will work well because I know little about prometheus.
Please test it.
Thanks.

Latest commit: 2f9434f

[towolf]

@vozlt I've tested this and it seems pretty good. It seems to make https://github.com/hnlq715/nginx-vts-exporter entirely unnecessary. It compared both and replaced just the metric names and it was basically a drop-in replacement. Only some connection states in nginx_vts_main_connections seem to differ:

Separate VTS exporter

Native VTS

Now, for the future, request times sorted into histogram buckets or percentiles would be great.

[sysulq]

nice job，maybe i should encouter this feature in nginx-vts-exporter project

[SuperQ]

@towolf Looks good. The connections graph doesn't seem far off, I'm guessing it's a difference of polling jitter. What scrape interval are you using there? Maybe try speeding it up to something like 5-10s to get higher resolution on the gauge. Gauges are inherently sensitive to polling jitter.

It would be useful to have some counters, like nginx_connections_total.

I sent a couple of PRs to do some best practices cleanup, but otherwise this looks pretty good.

[discordianfish]

Now, for the future, request times sorted into histogram buckets or percentiles would be great.
@vozlt Is that something you would be willing to add? I guess it's a bit more complicated and the bucket sizes would need to be configurable. Here are a few words on how it looks like: https://prometheus.io/docs/concepts/metric_types/#histogram

But basically the vts module would have to observe the duration of each request and add it to the right bucket. Pre-sampled averages and cut-off samples like exposed right now are unfortunately not enough to get any meaningful averages (see sysulq/nginx-vts-exporter#43 (comment))

[vozlt]

@discordianfish I will check out the histogram of prometheus soon.
Thanks.

[vozlt]

The metric type of histogram in format/prometheus has added.
See the directive vhost_traffic_status_histogram_buckets for details.
I quickly added the feature and did not test it much.
So it is necessary to verify that I have implemented the function correctly according to prometheus histogram metric type.
Please test it.
Thanks.

Latest commit: 2fd3ae4

[towolf]

Tested this again, and from what I can see at first glance, this works as it should.

Grafana can directly present histogram data as is as a heatmap. And Prometheus can transform histogram buckets into percentiles using this approximative function.

I only have sparse data at the moment:

Pretty great, thanks.

[discordianfish]

This is great, I'll test it and report back!

[draskolnikova]

@vozlt @hnlq715

I've seen upstreame state (down|up) in json format, would you mind to expose that in prometheus format?

[yangboyd]

@towolf Would you please share your grafana dashboard?
Thanks a lot!

Tested this again, and from what I can see at first glance, this works as it should.

Grafana can directly present histogram data as is as a heatmap. And Prometheus can transform histogram buckets into percentiles using this approximative function.

I only have sparse data at the moment:

Pretty great, thanks.

[towolf]

@yangboyd sure, here you go

But this dashboard is pretty specific for our old Kubernetes ingress setup, which at the time was a stand-alone thing.

But you can check it out, if you want.