Metrics

Curent Architecture

At the moment, our system uses Loki, Promtail, and Grafana to handle logging for the application. Promtail runs next to the application and collects logs directly from stdout or log files, then sends them to Loki. Loki stores these logs efficiently by indexing only the labels and keeping the actual log content compressed. This makes it good for handling large amounts of logs without using too much storage. Grafana sits on top and allows us to view and query logs using LogQL. While this setup works well for logging, it is not designed for application metrics. Loki can generate some metrics from logs, but this approach is slow and not reliable for real-time monitoring.

We are also starting to introduce OpenTelemetry into the system. With OpenTelemetry, the application can send structured logs that follow a consistent format. These logs go through the OpenTelemetry Collector, which can then forward them to Loki using a Loki exporter. This gives us a more standardized way to collect logs and prepares us for a more unified observability stack. The overall flow becomes: the application sends OpenTelemetry-formatted logs, the Collector processes and routes them, Loki stores them, and Grafana visualizes them. This keeps our existing logging setup but makes it more flexible and future-proof.

Prometheus

However, when it comes to metrics, Loki simply isn’t the right tool. Metrics need a system that is built for time-series data, fast queries, and accurate real-time alerting. This is why Prometheus is a better choice. Prometheus is designed specifically for metrics and works very well for things like request rates, error counts, latency, CPU usage, and other performance indicators. It is much faster and more reliable for this type of data, and it integrates cleanly with Alertmanager for handling alerts. Compared to deriving metrics from logs, using Prometheus is a more appropriate and efficient solution.

Prometheus also works well with OpenTelemetry. The application records metrics using the OpenTelemetry SDK, and these metrics are sent to the OpenTelemetry Collector. The Collector exposes them in a format that Prometheus can scrape. Prometheus then stores and queries the metrics, while Grafana is used to build dashboards and visualizations. In this setup, OpenTelemetry becomes the standard way we instrument our application, and Prometheus becomes the main system for metric storage and analysis.

In summary, Loki will continue to handle logs, now with the added structure provided by OpenTelemetry, while Prometheus will take over metrics using the same instrumentation approach. Grafana remains the visualization layer for both. This gives us a cleaner and more scalable observability setup that uses each tool for what it does best, while keeping everything aligned under the OpenTelemetry standard.

What VeloSim Tracks

VeloSim metrics fall into three categories: custom instrumentation, automatic Prometheus client metrics, and derived Grafana calculations.

Custom VeloSim Metrics (Explicitly Instrumented)

API Endpoint Metrics (PR #483):

• http_requests_total: Counter tracking total HTTP requests
• http_request_duration_seconds: Histogram of request latencies
• Labels: method, path, status
• Implementation: back/middleware/metrics_middleware.py

Simulation Metrics (PR #322):

• simulation.startup.time: Histogram of simulation startup duration (seconds)
• Implementation: back/core/simulation_startup_monitor.py

Frontend Action Metrics (PR #484):

• frontend.action.total: Counter of frontend user actions
• Labels: action type, entity type, user context
• Implementation: back/services/frontend_log_service.py

Automatic Prometheus Python Client Metrics

These are collected automatically by the Prometheus Python client without custom instrumentation:

• process_cpu_seconds_total: Total CPU time consumed by the process
• process_resident_memory_bytes: Resident memory size (RSS)
• process_virtual_memory_bytes: Virtual memory size (VMS)
• process_open_fds: Number of open file descriptors
• process_start_time_seconds: Process start time

Derived Metrics (Calculated in Grafana)

These are computed from raw metrics using PromQL queries:

• CPU Usage %: rate(process_cpu_seconds_total[1m]) * 100
• Request Rate: rate(http_requests_total[5m])
• Error Rate: rate(http_requests_total{status!~"2.."}[5m])
• p95 Latency: histogram_quantile(0.95, sum by (path, le) (rate(http_request_duration_seconds_bucket[5m])))
• Memory Growth Rate: deriv(process_resident_memory_bytes[5m])

Grafana Dashboards

VeloSim API Metrics (velosim_api_metrics.json):

• Request rate by endpoint
• p95 response time by endpoint
• Error rate by status code

VeloSim Python Backend (python-performance.json):

• CPU usage over time and current gauge
• Memory usage (resident/virtual) over time and current gauge
• Memory growth rate (leak detection)

Frontend Logs (frontend-logs.json):

• Log stream from Loki
• Frontend error tracking

Integration

Metrics are exposed at /metrics endpoint in Prometheus format. The Grafana instance queries Prometheus for visualization.

How to access metrics on production

1. Open VSCode and enter this command in the terminal (root directory): ssh -L 3001:localhost:3001 -L 3100:localhost:3100 -L 9090:localhost:9090 your_ENCS_username@velosim.app -p 2222 (note: You need to replace your_ENCS_username in the command)
2. Once successfully logged in, visit the Grafana page at 127.0.0.1:3001
3. Username: admin, Password: admin to log in.
4. On the left-side of the web page click on Dashboards. Here you will see the available dashboards to see Logs & Metrics originating from the Velosim app.

Examples

Here are example functions showing how we use Prometheus and OpenTelemetry to collect metrics for the BIXI simulation:

1. Setup and Meter Initialization

import time
from typing import Dict, List, Tuple
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.prometheus import PrometheusMetricReader

# Set up OpenTelemetry with Prometheus exporter
prom_reader = PrometheusMetricReader()
metrics.set_meter_provider(MeterProvider(metric_readers=[prom_reader]))
meter = metrics.get_meter("bixi-simulation-metrics")

# Storage for demonstration purposes
_demo_metrics: Dict[str, List[Tuple[Dict[str, str], float]]] = {
    "counters": [],
    "histograms": [],
}

2. Counter Metric

# Define a counter for simulation starts
simulation_start_counter = meter.create_counter(
    name="simulation_start_total",
    description="Number of BIXI simulation runs started",
)

def example_simulation_start_counter() -> None:
    labels = {"simulation_type": "bixi", "neighborhood": "downtown", "environment": "prod"}
    simulation_start_counter.add(1, labels)
    _demo_metrics["counters"].append((labels, 1))

3. Histogram Metric

# Define a histogram for simulation startup durations
simulation_start_duration = meter.create_histogram(
    name="simulation_start_duration_ms",
    description="Startup duration for BIXI simulation runs",
    unit="ms",
)

def example_simulation_start_timing() -> None:
    start = time.time()
    time.sleep(0.18)  # simulate startup time
    duration_ms = (time.time() - start) * 1000
    labels = {
        "simulation_type": "bixi",
        "neighborhood": "plateau",
        "environment": "staging",
        "status": "started"
    }
    simulation_start_duration.record(duration_ms, labels)
    _demo_metrics["histograms"].append((labels, duration_ms))

4. Full Simulation Metrics

def example_full_simulation_metrics() -> Dict[str, float]:
    t0 = time.time()
    time.sleep(0.20)  # simulate full simulation startup
    duration_ms = (time.time() - t0) * 1000
    labels = {
        "simulation_type": "bixi",
        "neighborhood": "montreal_centre",
        "environment": "prod",
        "status": "started"
    }
    simulation_start_counter.add(1, labels)
    simulation_start_duration.record(duration_ms, labels)
    _demo_metrics["counters"].append((labels, 1))
    _demo_metrics["histograms"].append((labels, duration_ms))
    return {"startup_time_ms": round(duration_ms, 2)}

5. Example /metrics endpoint

def example_metrics_endpoint() -> str:
    lines = []
    # Counters
    lines.append("# HELP simulation_start_total Number of simulations started")
    lines.append("# TYPE simulation_start_total counter")
    for labels, value in _demo_metrics["counters"]:
        label_str = ",".join(f'{k}="{v}"' for k, v in labels.items())
        lines.append(f"simulation_start_total{{{label_str}}} {value}")

    # Histograms
    lines.append("# HELP simulation_start_duration_ms Startup duration for simulation runs")
    lines.append("# TYPE simulation_start_duration_ms histogram")
    for labels, value in _demo_metrics["histograms"]:
        label_str = ",".join(f'{k}="{v}"' for k, v in labels.items())
        buckets = [50, 100, 200, float("inf")]
        count = 0
        for b in buckets:
            if value <= b:
                count += 1
            lines.append(f'simulation_start_duration_ms_bucket{{{label_str},le="{b}"}} {count}')
        lines.append(f"simulation_start_duration_ms_sum{{{label_str}}} {value:.2f}")
        lines.append(f"simulation_start_duration_ms_count{{{label_str}}} 1")
    return "\n".join(lines)