PoC for monitoring python application with Prometheus and EFK
- Learning how to write python (Flask) applications with Prometheus metrics exporting
- Creating application which simulates some failures and warnings
- Monitoring applications and containerized infrastructure with Prometheus
- Collecting logs with EFK
- Configuring alerting
MonApp is a Flask application created for learning how to build monitoring systems.
Every 10 seconds the application prints a log entry in console.
Mon, 16 Aug 2021 15:04:12 - INFO - dynauend whoilt wuoy
Mon, 16 Aug 2021 15:04:22 - INFO - gnaiantz psycheerly viaob
Mon, 16 Aug 2021 15:04:32 - ERROR - twoand twouc snoarr
Mon, 16 Aug 2021 15:04:42 - WARNING - sriaontz gills cycloech
Mon, 16 Aug 2021 15:04:51 - INFO - hyiaoght mckews blaiarry
Mon, 16 Aug 2021 15:05:02 - WARNING - wauefy dioutch mcciur
Mon, 16 Aug 2021 15:05:12 - INFO - schwoink psychong phantz
Mon, 16 Aug 2021 15:05:22 - INFO - groiats hiidy kosm
http://{IP}:5000/metrics
The application exposes some metrics about warehouses effectiveness:
# HELP monapp_position_in_stock Position in stock (1000 tons)
# TYPE monapp_position_in_stock gauge
monapp_position_in_stock{position="Rice"} 15.3699
monapp_position_in_stock{position="Corn"} 11.3835
http://{IP}:5000/alert
MonApp can work as a web hook for your AlertManager. It will show requests from AlertManager amd print them in console in raw format.
There are two URIs for testing CPU and RAM load.
http://{IP}:5000/cpu_task
This task will create a thread which will use as much CPU resources as possible during next 30 seconds.
http://{IP}:5000/ram_task
This task will create a thread which will use as much RAM as possible until the application is killed by docker.
- Prometheus and AlertManager
- ElasticSearch
- Kibana
- Grafana
- Fluentd
- cAdvisor
- Docker
http://{IP}:5601/
Logs are collected by Fluentd using Docker fluentd driver. Fluentd filters out only critical, error and warning messages and redirects them to ElasticSearch. All other messages will be ignored.
Collected logs are exposed using Kibana. Dashboards for Kibana you can find here (they will not be uploaded automatically during deployment).
http://{IP}:3000/
All the metrics are collected by Prometheus and exposed in Grafana. Find specially created dashboards in there.
Alertmanager is configured to fire up an alert if it finds that monapp_position_in_stock metric is below 10 or greater than 20. In this case it will send a request to a webhook http://{IP}:5000/alert.
For infrastructure monitoring cAdvisor container is added and configured. All metrics from cAdvisor are collected by Prometheus`` and exposed in Grafana``` dashboards.
MonApp has some mocking tools which can help you to test you monitoring infrastructure.