-
Notifications
You must be signed in to change notification settings - Fork 0
VeloSim Observability & Performance Insights
VeloSim uses a monitoring stack to collect, store, and visualize application data in real-time. This infrastructure serves two purposes: identifying performance bottlenecks in the application, and providing visibility into how the system behaves under real usage.
The stack consists of:
- Prometheus - collects and stores time-series metrics (request counts, latency, CPU, memory)
- Loki - aggregates and stores application logs
- Promtail - ships log files from the application to Loki
- Grafana - visualizes metrics and logs through dashboards
Every API request is automatically instrumented by the MetricsMiddleware. Two metrics are recorded for every request:
-
http_requests_total(Counter) - The total number of HTTP requests, labeled by method, path, and status code. -
http_request_duration_seconds(Histogram) - How long each request took to process, labeled by method, path, and status code.
These are recorded automatically - no per-endpoint code changes are needed. The middleware captures the route path template (e.g., /api/v1/simulation/{id}/playbackSpeed), the HTTP method, the response status code, and the duration using time.perf_counter().
The Python prometheus_client library automatically exposes process-level metrics:
-
process_cpu_seconds_total- Cumulative CPU time consumed by the backend process -
process_resident_memory_bytes- Physical RAM used by the backend -
process_virtual_memory_bytes- Total virtual memory allocated
A custom histogram simulation.startup.time tracks the total time from when a user sends an initialization request to when the first simulation frame is emitted. This measures the real delay operators experience when starting a simulation.
When the frontend sends log entries with context information, a counter frontend.action.total is incremented with labels for the action type and user ID. This tracks which frontend features operators interact with.
The backend writes structured logs to a file (logs.txt) which Promtail ships to Loki. Logs include module name, log level, and message content. These can be queried in Grafana using LogQL (e.g., {job="velosim_app"} |= "ERROR").
| Panel | Query | What It Shows |
|---|---|---|
| Request Rate by Endpoint | sum by (path) (rate(http_requests_total[5m])) |
How many requests each endpoint receives per second. Identifies the most-used features. |
| 95th Percentile Latency | histogram_quantile(0.95, sum by (path, le) (rate(http_request_duration_seconds_bucket[5m]))) |
95% of requests to each endpoint complete faster than this value. Identifies slow endpoints. |
| Error Rate by Endpoint | sum by (path) (rate(http_requests_total{status!~"2.."}[5m])) |
Shows how often each endpoint returns errors (non-2xx responses) per second, helping identify unreliable features. |
| Requests by Status Code | sum by (status) (rate(http_requests_total[5m])) |
Overall distribution of response codes (200, 400, 500, etc.). |
| Panel | Query | What It Shows |
|---|---|---|
| CPU Usage (%) | rate(process_cpu_seconds_total{job="velosim-backend"}[1m]) * 100 |
CPU utilization over time. Spikes indicate compute-heavy operations. |
| Memory Usage |
process_resident_memory_bytes + process_virtual_memory_bytes
|
RAM and virtual memory usage. |
| CPU Gauge | Same as above | Current CPU as a gauge. Green <50%, Yellow 50-80%, Red >80%. |
| Memory Gauge | process_resident_memory_bytes |
Current RAM. Green <512MB, Yellow <1GB, Red >1GB. |
| Memory Growth Rate | deriv(process_resident_memory_bytes{job="velosim-backend"}[5m]) |
Rate of memory increase in bytes/sec. A sustained positive value indicates a memory leak. |
The 95th percentile latency panel shows which API endpoints take the longest to respond. If simulation initialization takes 4 seconds but station queries take 50ms, we know where optimization effort should go. Operators experience these delays directly. A slow endpoint means a laggy UI.
Action: Prioritize optimization on the slowest endpoints that users interact with frequently.
Request rate by endpoint reveals which features operators actually use. If the replay endpoint gets heavy traffic but branching is rarely called, that tells us where to focus development. Features with low traffic might not need optimization, while high-traffic features need to be fast and reliable.
Action: Invest development time proportional to how much each feature is used.
The error rate panel shows which endpoints return non-2xx responses most often. A 5% error rate on a critical endpoint like simulation playback control means operators are regularly hitting failures.
Action: Investigate and fix endpoints with high error rates before adding new features.
By monitoring these metrics across releases, we can detect when a code change makes something slower or less reliable. If a deployment doubles the p95 latency on the simulation endpoint, it shows up immediately in the dashboard rather than waiting for user complaints.
Action: Check dashboards after each deployment to catch regressions early.
CPU and memory metrics show how close we are to the server's limits under normal usage. The memory growth rate panel specifically catches memory leaks. If memory climbs steadily during long-running simulations and never drops, the server will eventually degrade.
Action: Monitor memory growth during extended simulation runs. Investigate if memory consistently increases without releasing.
When latency spikes, CPU and memory panels viewed alongside the API metrics reveal the root cause. If latency spikes with high CPU, the problem is due to a lot of computations being performed. If latency spikes with low CPU, the bottleneck is elsewhere (database, OSRM routing, I/O).
Action: Use correlated panels to determine whether performance issues are CPU-bound, memory-bound, or I/O-bound before optimizing.
Every API request is automatically instrumented by the MetricsMiddleware. Two metrics are recorded for every request:
-
http_requests_total(Counter) - The total number of HTTP requests, labeled by method, path, and status code. -
http_request_duration_seconds(Histogram) - How long each request took to process, labeled by method, path, and status code.
These are recorded automatically, no per-endpoint code changes are needed. The middleware captures the route path template (e.g., /api/v1/simulation/{id}/playbackSpeed), the HTTP method, the response status code, and the duration.
The Python prometheus_client library automatically exposes process-level metrics:
-
process_cpu_seconds_total- Cumulative CPU time consumed by the backend process -
process_resident_memory_bytes- Physical RAM used by the backend -
process_virtual_memory_bytes- Total virtual memory allocated
A custom histogram simulation.startup.time tracks the total time from when a user sends an initialization request to when the first simulation frame is emitted. This measures the real delay operators experience when starting a simulation.
When the frontend sends log entries with context information, a counter frontend.action.total is incremented with labels for the action type and user ID. This tracks which frontend features operators interact with.
The backend writes structured logs to a file (logs.txt) which Promtail ships to Loki. Logs include module name, log level, and message content. These can be queried in Grafana using LogQL (e.g., {job="velosim_app"} |= "ERROR").
| Panel | Query | What It Shows |
|---|---|---|
| Request Rate by Endpoint | sum by (path) (rate(http_requests_total[5m])) |
How many requests each endpoint receives per second. Identifies the most-used features. |
| 95th Percentile Latency | histogram_quantile(0.95, sum by (path, le) (rate(http_request_duration_seconds_bucket[5m]))) |
95% of requests to each endpoint complete faster than this value. Identifies slow endpoints. |
| Error Rate by Endpoint | sum by (path) (rate(http_requests_total{status!~"2.."}[5m])) |
Shows how often each endpoint returns errors (non-2xx responses) per second, helping identify unreliable features. |
| Requests by Status Code | sum by (status) (rate(http_requests_total[5m])) |
Overall distribution of response codes (200, 400, 500, etc.). |
| Panel | Query | What It Shows |
|---|---|---|
| CPU Usage (%) | rate(process_cpu_seconds_total{job="velosim-backend"}[1m]) * 100 |
CPU utilization over time. Spikes indicate compute-heavy operations. |
| Memory Usage |
process_resident_memory_bytes + process_virtual_memory_bytes
|
RAM and virtual memory usage. |
| CPU Gauge | Same as above | Current CPU as a gauge. Green <50%, Yellow 50-80%, Red >80%. |
| Memory Gauge | process_resident_memory_bytes |
Current RAM. Green <512MB, Yellow <1GB, Red >1GB. |
| Memory Growth Rate | deriv(process_resident_memory_bytes{job="velosim-backend"}[5m]) |
Rate of memory increase in bytes/sec. A sustained positive value indicates a memory leak. |
The 95th percentile latency panel shows which API endpoints take the longest to respond. If simulation initialization takes 4 seconds but station queries take 50ms, we know where optimization effort should go. Operators experience these delays directly. A slow endpoint means a laggy UI.
Action: Prioritize optimization on the slowest endpoints that users interact with frequently.
Request rate by endpoint reveals which features operators actually use. If the replay endpoint gets heavy traffic but branching is rarely called, that tells us where to focus development. Features with low traffic might not need optimization, while high-traffic features need to be fast and reliable.
Action: Invest development time proportional to how much each feature is used.
The error rate panel shows which endpoints return non-2xx responses most often. A 5% error rate on a critical endpoint like simulation playback control means operators are regularly hitting failures.
Action: Investigate and fix endpoints with high error rates before adding new features.
By monitoring these metrics across releases, we can detect when a code change makes something slower or less reliable. If a deployment doubles the p95 latency on the simulation endpoint, it shows up immediately in the dashboard rather than waiting for user complaints.
Action: Check dashboards after each deployment to catch regressions early.
CPU and memory metrics show how close we are to the server's limits under normal usage. The memory growth rate panel specifically catches memory leaks. If memory climbs steadily during long-running simulations and never drops, the server will eventually degrade.
Action: Monitor memory growth during extended simulation runs. Investigate if memory consistently increases without releasing.
When latency spikes, CPU and memory panels viewed alongside the API metrics reveal the root cause. If latency spikes with high CPU, the problem is due to a lot of computations being performed. If latency spikes with low CPU, the bottleneck is elsewhere (database, OSRM routing, I/O).
Action: Look at the CPU, memory, and API metrics together to figure out whether performance issues are caused by computation, memory usage, or I/O before making optimizations.
Simulation Meeting Minutes
Frontend Meeting Minutes
Backend Meeting Minutes
Risks
User Consent and End-User License Agreement
Legal and Ethical Issues
Economic
Budget
Personas
Diversity Statement
Overall Architecture and Class Diagrams
Infrastructure and Tools
Name Conventions
Testing Plan and Continuous Integration
Security
Performance
Deployment Plan and Infrastructure
Missing Knowledge and Independent Learning
Glossary
Mockups
UI Evolution
Logging
Metrics
VeloSim Observability & Performance Insights
User Manual
Usability Tests