More detailed metrics

[keis]

It would be great if traefik could provide some more detailed metrics, e.g

• Response time quantile to avoid skew from one or two outliers. (like how prometheus does it)
• Breakdown by backend
  • request count
  • response time

[thawkins]

Definatly +1 on this....

I would also add response status to that list.

[keis]

Most of the heavy lifting for this is done by the logger middleware now. It should be pretty simple to add some prometheus counters to that. Not sure if that's the best place for doing it or if prometheus is the right tool. I do quite like it though :)

[samber]

I think most interesting metrics are:

• traefik build id
• per entrypoint:
  • Number of active connections
• per backend type (consul catalog, file, etcd, kubernetes...):
  • Number of successful backend updates
  • Number of failed backend updates
  • number catched updates
  • number effective reload
• per service:
  • circuit breaker:
    • number of backends available
    • number failing backends
  • frontend:
    • response time (including Traefik processing)
    • number successful requests (1xx, 2xx, 3xx)
    • number failing requests (4xx, 5xx)
    • request size
    • response size
  • per backend:
    • response time (excluding Traefik processing)
    • number successful requests (1xx, 2xx, 3xx)
    • number failing requests (4xx, 5xx)
    • retry count
    • circuit breaker:
      • is available (bool)
      • expression (example: NetworkErrorRatio() > 0.5)
      • number network error

Prometheus is probably the more "trendly" time series db. IMO, that's why we should integrate a /metrics route in the web provider, using the Prometheus data format.

Prometheus can aggregate backends stats to get avg or sum of each metrics, then adding "total" counts would only be used for /healthrequests.

[thawkins]

+1 on prometheus integration, prometheus will be out of the box on the next release of kubenetes. Having good integration would be excellent.

[samber]

Last week, I made a standalone Prometheus exporter: https://github.com/iadvize/traefik-exporter

I will combine that work with the Traefik binary very soon.

As an example of limitation of using an external exporter: we cannot use Prometheus summaries to mesure the duration of 1% slowest requests.

[therc]

It would be great to also have the lists and rates by top N client IPs or some other fields defined in the configuration (top Host:, Authentication: headers). Maybe even have an endpoint to add IPs and headers to track manually, even if they're not in the top N. This would allow user to run a DDOS detection/mitigation system or a throttler alongside Traefik. I doubt most people will want either of those built-in or that it would meet all their needs.

[brian-brazil]

Top N isn't really well suited to Prometheus, as it'll cause too much label churn - particularly on this type of system which may be talked to by the entire internet.

All other mentioned use cases are perfect for Prometheus, maybe add last update time too? The exposition format is open and there's parsers currently available in Go and Python so it's possible to integrate into other monitoring systems too.

[tlvenn]

HAProxy has a pretty useful dashboard when it comes to metrics, probably a good path to follow

http://demo.haproxy.org/

Would it very nice to have something similar out of the box with Traefik.

[arehmandev]

Looks like metrics has now been implementedt:

175659a

[enxebre]

this is partially covered now by #1022 and #1042. Which should serve as a foundation for adding more detailed metrics.

[timoreimann]

I'm going to close this issue as basic support has shipped.

The list in #156 (comment) and @brian-brazil's suggestion in #156 (comment) regarding uptime should provide as a great basis for future additions. Please file dedicated issues for additional metrics.