Feature/prometheus skipper metrics

[szuecs]

This is a breaking change please make sure we only roll this out to dev and not to alpha and beta, before all important zmon checks and alerts have replacements.

This feature enables us to have better aggregated metrics, that are not as complicated as before.
It should also fix the problem of skipper-ingress pod restart will loose all metrics and the aggregation looks awkward.

I am not sure if we should add more prometheus replicas and use a service to make it more resilient to cluster updates, etc.

[mikkeloscar]

We need a compliant image here.

[mikkeloscar]

@szuecs can we switch one of the dev clusters to this branch and prepare the checks before we start rolling this into dev and further?

[szuecs]

@mikkeloscar sure!

[szuecs]

volume mount issue: kubernetes/kubernetes#2630
Prometheus is using nobody as user inside the container and kubernetes does not support it.

This is a fix for the volume problem

  securityContext:
    runAsUser: 65534
    fsGroup: 65534

[mikkeloscar]

A comment on why this is needed would be good for the future :)

I guess 65534 -> nobody?

[szuecs]

Yes, but I will not change it, because if the nobody user get 42 as uid then this container can not read and write data anymore, because of uid miss match (media has 65534 and uid 42), which should be in theory changed, but better to be sure.

[szuecs]

Current state is that I found solutions for all metrics we want to provide.

- [X] http5xx_count           round(http5xx_errors_per_sec * 60, 0)
- [X] http5xx_errors_per_sec  (6) sum(rate(skipper_serve_host_duration_seconds_count{}[1m])) by (host, code)
- [X] http5xx_rate            min(round(http5xx_count * 100. / requests_count, 4), 100.) if requests_count > 0 else 0
- [X] http4xx_count           round(http4xx_errors_per_sec * 60, 0)
- [X] http4xx_errors_per_sec  (6) sum(rate(skipper_serve_host_duration_seconds_count{}[1m])) by (host, code)
- [X] http4xx_rate            min(round(http4xx_count * 100. / requests_count, 4), 100.) if requests_count > 0 else 0
- [X] requests_count          round(requests_per_sec * 60, 0)
- [X] requests_per_sec        (7) or sum_code(6) - sum(rate(skipper_serve_host_duration_seconds_count{}[1m])) by (host)
- [X] latency_ms              (5) sum(rate(skipper_serve_host_duration_seconds_sum{}[1m])) by (host) / sum(rate(skipper_serve_host_duration_seconds_count{}[1m])) by (host)
- [X] latency_p90_ms          (4) histogram_quantile(0.90, sum(rate(skipper_serve_host_duration_seconds_bucket{}[1m])) by (le, host) )
- [X] latency_p95_ms          (3) histogram_quantile(0.95, sum(rate(skipper_serve_host_duration_seconds_bucket{}[1m])) by (le, host) )
- [X] latency_p99_ms          (2) histogram_quantile(0.99, sum(rate(skipper_serve_host_duration_seconds_bucket{}[1m])) by (le, host) )
- [X] latency_p999_ms         (1) histogram_quantile(0.999, sum(rate(skipper_serve_host_duration_seconds_bucket{}[1m])) by (le, host) )
# zmon can calculate on prom data
- [X] http_error_per_sec          # http4xx_errors_per_sec + http5xx_errors_per_sec
- [X] http_error_count            # http4xx_count + http5xx_count
- [X] http_error_rate             # http4xx_rate + http5xx_rate

[szuecs]

zalando/skipper#569 implemented a soft migration path for enabling users to have both versions of metrics exposed. We can do now the soft migration from codahale to prometheus.

[szuecs]

👍

[mikkeloscar]

This should now be apps/v1 (since v1.9)

[mikkeloscar]

We should have the version label here to be consistent with the other manifests. :)

[mikkeloscar]

Also version label here :)

[szuecs]

comments will be fixed in a separate PR

[mikkeloscar]

👍