-
Notifications
You must be signed in to change notification settings - Fork 0
Monitoring
We want to remotely monitor the health, activity, and related metrics of
- back-end application code
- web server, e.g. Nginx (production) or Uvicorn (development)
- database
- front-end
For the first three items, we use Prometheus for metrics and Grafana for visual display of the data.
| Component | Monitor | URL |
|---|---|---|
| FastAPI WS |
prometheus-fastapi-instrumentor + custom code in main.py
|
http://localhost:8000/metrics |
| Nginx |
/nginx_status location invoked Nginx's stub_status module for metrics. For security, accessible only inside Docker network |
http://nginx:8080/nginx_status |
| Nginx Exporter |
nginx-exporter collects and adapts Nginx data for Prometheus |
http://nginx-exporter:9113/metrics |
| Prometheus Server | Collects & records metrics. Paths: /metrics, /targets, /query
|
http://localhost:9090 |
| Grafana UI | Grafana visualizer for Prometheus and other services | http://localhost:4000 |
Note that Docker uses an internal DNS to resolve container names like nginx to IP addresses.
Use docker network ls to discover network names, docker network inspect homelog_default to discover container IP addresses.
To verify a container name (nginx-exporter) is being resolved to an internal IP address:
docker exec -it nginx nslookup nginx-exporterAs of this writing, the relevant files and directories are:
project/
├── docker-compose.yml Configure containers, exposed ports, and volumes
├── backend/
│ ├── Dockerfile FastAPI web service + Uvicorn
│ └── main.py Configure and expose metrics for Prometheus
├── nginx/ Nginx proxies the backend web service
│ └── nginx.conf Exposes metrics (in the docker network) at path `/nginx_status`
└── monitor/
└── prometheus.yml
Prometheus collects metrics data from various sources. These pages describe how to expose metrics from different sources:
- Monitoring FastAPI
- Monitoring Nginx
- TODO Monitor Postgres Database
Defining a Grafana dashboard from scratch is time-consuming. You can import dashboards by ID from Grafanalabs and connect them to your own data source.
Discover, explore, and download dashboards: https://grafana.com/grafana/dashboards/
Nginx Dashboards for nginx-prometheus-exporter are fairly standard:
| Dashboard ID | Description |
|---|---|
| 12708 | Official Nginx exporter dashboard, designed for the exporter, 2020-07-26 |
| 12452 | Another dashboard designed explicitly for nginx-prometheus-exporter, 2022-11-21 |