Skip to content

Monitoring

Jump to bottom
James Brucker edited this page Nov 27, 2025 · 10 revisions

We want to remotely monitor the health, activity, and related metrics of

  • back-end application code
  • web server, e.g. Nginx (production) or Uvicorn (development)
  • database
  • front-end

For the first three items, the initial implementation will use Prometheus for metrics and Grafana for visual display of the data.

Monitoring Components and URLs

Component Monitor URL
FastAPI WS prometheus-fastapi-instrumentor + custom code in main.py http://localhost:8080/metrics
Nginx /nginx_status location, accessible only inside Docker network http://nginx:8080/nginx_status
Nginx Exporter nginx-exporter running in a separate container exposed port 9113 http://localhost:9113/metrics
Prometheus Server Collects ("scrapes") metrics. Paths: /metrics, /targets, /query http://localhost:9090
Grafana UI Grafana visualizer for Prometheus. http://localhost:4000

Note that Docker uses an internal DNS to resolve container names like nginx to IP addresses. Use docker network ls to discover network names, docker network inspect homelog_default to discover container IP addresses. This is needed in Nginx config file to limit IP addresses that can access Nginx statistics.

As of this writing, the relevant files and directories are:

project/
├── docker-compose.yml    Configure containers, exposed ports, and volumes
├── backend/
│   ├── Dockerfile        FastAPI web service + Uvicorn
│   └── main.py           Configure and expose metrics for Prometheus
├── nginx/                Nginx proxies the backend web service
│   └── nginx.conf        Exposes metrics at path `/nginx_status`
└── monitor/
    └── prometheus.yml

Grafana Dashboards for Visualizing Data

Defining a Grafana dashboard from scratch is time-consuming. You can import dashboards by ID from Grafanalabs and connect them to your own data source.

Discover, explore, and download dashboards: https://grafana.com/grafana/dashboards/

Nginx Dashboards for nginx-prometheus-exporter are fairly standard:

Dashboard ID Description
12708 Official Nginx exporter dashboard, designed for the exporter, 2020-07-26
12452 Another dashboard designed explicitly for nginx-prometheus-exporter, 2022-11-21

Instrument FastAPI Back-end

Method 1: Use prometheus-fastapi-instrumentator

This method is recommended by Deepseek :-) as it provides "comprehensive metrics out of the box with minimal configuration".

  1. Use pip to install the prometheus-fastapi-instrumentator package.

  2. In main.py (where the FastAPI object is created), add:

    from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

# Instrument the application
instrumentator = Instrumentator( <options> )
instrucmentor.instrument(app)
instrumentator.expose(app)

3. (Optional) Advanced configuration.
   ```python
   from fastapi import FastAPI, Request
   from prometheus_fastapi_instrumentator import Instrumentator, metrics

   app = FastAPI()

   # Custom instrumentation with more metrics
   instrumentator = Instrumentator(
       should_group_status_codes=True,
       should_ignore_untemplated=True,
       should_respect_env_var=True,
       should_instrument_requests_inprogress=True,
       excluded_handlers=["/metrics", "/health"],
       env_var_name="ENABLE_METRICS",
       inprogress_name="inprogress",
       inprogress_labels=True,
   )

   # Add default metrics
   instrumentator.add(
       metrics.request_size(
           should_include_handler=True,
           should_include_method=True,
           should_include_status=True,
       )
   ).add(
       metrics.response_size(
           should_include_handler=True,
           should_include_method=True,
           should_include_status=True,
       )
   ).add(
       metrics.latency(
           bucket_type="exponential",
           should_include_handler=True,
           should_include_method=True,
           should_include_status=True,
       )
   ).add(
       metrics.requests(
           should_include_handler=True,
           should_include_method=True,
           should_include_status=True,
       )
   )

   # Instrument the app
   instrumentator.instrument(app)
   instrumentator.expose(app)

Method 2: Use prometheus-client Directly

Step 1: Install the client. Use pip to install prometheus-client package.

Step 2: Create custom metrics

from fastapi import FastAPI, Request, Response
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST, Counter, Histogram, Gauge
import time

app = FastAPI()

# Define metrics
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP Requests',
    ['method', 'endpoint', 'status_code']
)

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP Request Latency',
    ['method', 'endpoint']
)

IN_PROGRESS = Gauge(
    'http_requests_in_progress',
    'HTTP Requests in Progress',
    ['method', 'endpoint']
)

@app.middleware("http")
async def monitor_requests(request: Request, call_next):
    method = request.method
    endpoint = request.url.path
    
    # Skip metrics endpoint
    if endpoint == "/metrics":
        return await call_next(request)
    
    IN_PROGRESS.labels(method=method, endpoint=endpoint).inc()
    start_time = time.time()
    
    try:
        response = await call_next(request)
        status_code = response.status_code
    except Exception as e:
        status_code = 500
        raise e
    finally:
        latency = time.time() - start_time
        REQUEST_LATENCY.labels(method=method, endpoint=endpoint).observe(latency)
        REQUEST_COUNT.labels(method=method, endpoint=endpoint, status_code=status_code).inc()
        IN_PROGRESS.labels(method=method, endpoint=endpoint).dec()
    
    return response

@app.get("/metrics")
async def metrics():
    return Response(
        content=generate_latest(),
        media_type=CONTENT_TYPE_LATEST
    )

Method 3: Complete Custom Implementation

This method also requires the prometheus-client package.

from fastapi import FastAPI, Request, Response, Depends
from prometheus_client import (
    generate_latest, CONTENT_TYPE_LATEST, Counter, Histogram, 
    Gauge, Summary, REGISTRY, CollectorRegistry
)
import time
from typing import Callable

app = FastAPI(title="My FastAPI App")

# Custom metrics registry
registry = CollectorRegistry()

# Application metrics
REQUEST_COUNT = Counter(
    'fastapi_requests_total',
    'Total count of HTTP requests',
    ['method', 'endpoint', 'status_code'],
    registry=registry
)

REQUEST_DURATION = Histogram(
    'fastapi_request_duration_seconds',
    'HTTP request duration in seconds',
    ['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0],
    registry=registry
)

REQUESTS_IN_PROGRESS = Gauge(
    'fastapi_requests_in_progress',
    'Number of HTTP requests in progress',
    ['method', 'endpoint'],
    registry=registry
)

ACTIVE_USERS = Gauge(
    'fastapi_active_users',
    'Number of active users',
    registry=registry
)

# Business logic metrics
ITEMS_CREATED = Counter(
    'fastapi_items_created_total',
    'Total number of items created',
    registry=registry
)

ERROR_COUNT = Counter(
    'fastapi_errors_total',
    'Total number of errors',
    ['error_type'],
    registry=registry
)

@app.middleware("http")
async def metrics_middleware(request: Request, call_next):
    method = request.method
    endpoint = request.url.path
    
    if endpoint == "/metrics":
        return await call_next(request)
    
    REQUESTS_IN_PROGRESS.labels(method=method, endpoint=endpoint).inc()
    start_time = time.time()
    
    try:
        response = await call_next(request)
        status_code = response.status_code
    except Exception as e:
        status_code = 500
        ERROR_COUNT.labels(error_type=type(e).__name__).inc()
        raise e
    finally:
        duration = time.time() - start_time
        REQUEST_DURATION.labels(method=method, endpoint=endpoint).observe(duration)
        REQUEST_COUNT.labels(method=method, endpoint=endpoint, status_code=status_code).inc()
        REQUESTS_IN_PROGRESS.labels(method=method, endpoint=endpoint).dec()
    
    return response

@app.get("/metrics")
async def metrics():
    return Response(
        content=generate_latest(registry),
        media_type=CONTENT_TYPE_LATEST
    )

Docker Configuration

Dockerfile

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000
EXPOSE 8001  # For metrics endpoint if on different port

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Docker Compose Configuration File

version: '3.8'
services:
  fastapi-app:
    build: .
    ports:
      - "8000:8000"
    environment:
      - PROMETHEUS_MULTIPROC_DIR=/tmp
    volumes:
      - /tmp:/tmp

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitor/prometheus.yml:/etc/prometheus/prometheus.yml

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'fastapi-app'
    static_configs:
      - targets: ['fastapi-app:8000']
    metrics_path: '/metrics'
    scrape_interval: 10s

Running and Testing

Start the application

uvicorn main:app --host 0.0.0.0 --port 8000

Test the metrics endpoint

curl http://localhost:8000/metrics

Generate some traffic

# Make some requests to generate metrics
curl http://localhost:8000/
curl http://localhost:8000/items/1
curl http://localhost:8000/health

Key Metrics Exported

With this setup, you'll get metrics like:

  • http_requests_total - Request counts by method, endpoint, status
  • http_request_duration_seconds - Request latency distribution
  • http_requests_in_progress - Current in-progress requests
  • Custom business metrics

Clone this wiki locally