Prometheus example alert rules

[DifferentialOrange]

Closes #43

Added Prometheus example alert rules:

• instance state (Prometheus up),
• Lua memory (warning and alert),
• arena/items limit warnings and alerts,
• high HTTP latency,
• too many HTTP 4xx responses (on a single instance or overall on cluster),
• HTTP 5xx errors,
• low router HTTP activity.

Alerts are separated into three groups. common is non-Tarantool alert on process work (Prometheus up). tarantool-common are general Tarantool alerts (Lua memory and slab_ratio) that can be applied to any Tarantool application. tarantool-business is a list of references on how you can monitor your business logic. One can base its alert rules on what's described there (because it's impossible to say if it is OK for your app to have 1000 RPS of 4xx errors or 0 requests on a router for an hour or not without knowing your app business logic beforehand). That's also the reason why I fixed job='example_project' in all tarantool-business alert rules while leaving common and tarantool-common alert rules process all possible Tarantool instances (if you have two different apps, they are likely to have different HTTP load and business logic, while 2 Gb Lua threshold is true for both of them).

Test Prometheus example alert rules with promtool.

The next step should be adding some documentation based on this example (here or in tarantool/metrics), but I think it should be a different PR.

[DifferentialOrange]

Example:

[DifferentialOrange]

The final note here is labels where I set severity: warning. Inspired by Google SRE book I have decided to use only 2 severity levels for alerting – warning and page. warning alerts should go to the ticketing system and you should react to these alerts during normal working days. page alerts are emergencies and can wake up on-call engineer – this type of alerts should be crafted carefully to avoid burnout. Alerts routing based on levels is managed by Alertmanager.

https://alex.dzyoba.com/blog/prometheus-alerts/