Add monitoring and stats for the raster server

[paulmelnikow]

It would be good to know how many requests are hitting the server, maybe get errors in Sentry, and have a frequent test that hits the server and ensures it's still responding with valid PNGs.

[platan]

I started to experiment with https://github.com/prometheus/blackbox_exporter. I installed it on metrics.shields.io and added two HTTP targets to Prometheus: https://shields.io/ and https://raster.shields.io/badge/foo-bar-blue.png. Checks are performed every 15 seconds (we can change it). This way we gather a lot of data. I also added https://grafana.com/grafana/dashboards/7587 to our Grafana.
A list of targets being checked is stored in: https://github.com/platan/metrics-shields-io-config/blob/3759b07de7fb1883506a3cededd2023944f00ca2/shields-io-metrics.yml#L63-L64
Final dashboard is available here: https://metrics.shields.io/d/xtkCtBkiz/prometheus-blackbox-exporter?orgId=1&refresh=10s

AFAIK it's not possible verify the response body using blackbox exporter. But we have stats with status code and content length of the response and we can use this data (expecting status code = 200 and content length = 972 (current response size of foo-bar-blue.png)).

[platan]

https://metrics.shields.io/d/xtkCtBkiz/prometheus-blackbox-exporter has stats for raster badge for the last two weeks. Average response time is 458 ms.

Today I've added probing of the most popular SVG badges: static_badge, npm_version, travis_build, npm_downloads, github_stars. Results can be seen at https://metrics.shields.io/d/xtkCtBkiz/prometheus-blackbox-exporter.

[paulmelnikow]

Nice! I'm guessing that request is hitting the Now CDN cache?

[platan]

I think you are right. Response times for new badges not requested before are 2x-3x longer than response times for badges requested several times.