VeloSim Observability & Performance Insights
VeloSim uses a monitoring stack to collect, store, and visualize application data in real-time. This infrastructure serves two purposes: identifying performance bottlenecks in the application, and providing visibility into how the system behaves under real usage.
The stack consists of:
- Prometheus - collects and stores time-series metrics (request counts, latency, CPU, memory)
- Loki - aggregates and stores application logs
- Promtail - ships log files from the application to Loki
- Grafana - visualizes metrics and logs through dashboards
Every API request is automatically instrumented by the MetricsMiddleware. Two metrics are recorded for every request:
-
http_requests_total(Counter) - The total number of HTTP requests, labeled by method, path, and status code. -
http_request_duration_seconds(Histogram) - How long each request took to process, labeled by method, path, and status code.
These are recorded automatically - no per-endpoint code changes are needed. The middleware captures the route path template (e.g., /api/v1/simulation/{id}/playbackSpeed), the HTTP method, the response status code, and the duration using time.perf_counter().
The Python prometheus_client library automatically exposes process-level metrics:
-
process_cpu_seconds_total- Cumulative CPU time consumed by the backend process -
process_resident_memory_bytes- Physical RAM used by the backend -
process_virtual_memory_bytes- Total virtual memory allocated
A custom histogram simulation.startup.time tracks the total time from when a user sends an initialization request to when the first simulation frame is emitted. This measures the real delay operators experience when starting a simulation.
When the frontend sends log entries with context information, a counter frontend.action.total is incremented with labels for the action type and user ID. This tracks which frontend features operators interact with.
The backend writes structured logs to a file (logs.txt) which Promtail ships to Loki. Logs include module name, log level, and message content. These can be queried in Grafana using LogQL (e.g., {job="velosim_app"} |= "ERROR").
| Panel | Query | What It Shows |
|---|---|---|
| Request Rate by Endpoint | sum by (path) (rate(http_requests_total[5m])) |
How many requests each endpoint receives per second. Identifies the most-used features. |
| 95th Percentile Latency | histogram_quantile(0.95, sum by (path, le) (rate(http_request_duration_seconds_bucket[5m]))) |
95% of requests to each endpoint complete faster than this value. Identifies slow endpoints. |
| Error Rate by Endpoint | sum by (path) (rate(http_requests_total{status!~"2.."}[5m])) |
Shows how often each endpoint returns non-success responses per second, helping identify unreliable features. |
| Requests by Status Code | sum by (status) (rate(http_requests_total[5m])) |
Overall distribution of response codes (200, 400, 500, etc.). |
| Panel | Query | What It Shows |
|---|---|---|
| CPU Usage (%) | rate(process_cpu_seconds_total{job="velosim-backend"}[1m]) * 100 |
CPU utilization over time. Spikes indicate compute-heavy operations. |
| Memory Usage |
process_resident_memory_bytes + process_virtual_memory_bytes
|
RAM and virtual memory usage. |
| CPU Gauge | Same as above | Current CPU as a gauge. Green <50%, Yellow 50-80%, Red >80%. |
| Memory Gauge | process_resident_memory_bytes |
Current RAM. Green <512MB, Yellow <1GB, Red >1GB. |
| Memory Growth Rate | deriv(process_resident_memory_bytes{job="velosim-backend"}[5m]) |
Rate of memory increase in bytes/sec. A sustained positive value indicates a memory leak. |
The 95th percentile latency panel shows which API endpoints take the longest to respond. If simulation initialization takes 4 seconds but station queries take 50ms, we know where optimization effort should go. Operators experience these delays directly. A slow endpoint means a laggy UI.
Action: Prioritize optimization on the slowest endpoints that users interact with frequently.
Request rate by endpoint reveals which features operators actually use. If the replay endpoint gets heavy traffic but branching is rarely called, that tells us where to focus development. Features with low traffic might not need optimization, while high-traffic features need to be fast and reliable.
Action: Invest development time proportional to how much each feature is used.
The error rate panel shows which endpoints return non-2xx responses most often. A 5% error rate on a critical endpoint like simulation playback control means operators are regularly hitting failures.
Action: Investigate and fix endpoints with high error rates before adding new features.
By monitoring these metrics across releases, we can detect when a code change makes something slower or less reliable. If a deployment doubles the p95 latency on the simulation endpoint, it shows up immediately in the dashboard rather than waiting for user complaints.
Action: Check dashboards after each deployment to catch regressions early.
CPU and memory metrics show how close we are to the server's limits under normal usage. The memory growth rate panel specifically catches memory leaks. If memory climbs steadily during long-running simulations and never drops, the server will eventually degrade.
Action: Monitor memory growth during extended simulation runs. Investigate if memory consistently increases without releasing.
When latency spikes, CPU and memory panels viewed alongside the API metrics reveal the root cause. If latency spikes with high CPU, the problem is due to a lot of computations being performed. If latency spikes with low CPU, the bottleneck is elsewhere (database, GraphHopper routing, I/O).
Action: Use correlated panels to determine whether performance issues are CPU-bound, memory-bound, or I/O-bound before optimizing.
The following sections show real data from VeloSim's production environment, demonstrating how our observability stack informs development decisions.
Understanding which endpoints take the longest to respond tells us where to focus optimization effort. Slow endpoints translate directly to a laggy user experience.
What this tells us:
- Simulation driver management endpoints (
/drivers/reorder,/drivers/reassign,/drivers/unassign,/drivers/assign) are the slowest, averaging 250-680ms. These involve database writes and simulation state updates. -
POST /api/v1/simulation/initializeaverages 130ms, which is the delay users experience when starting a new simulation. -
POST /api/token(login) takes 121ms due to Argon2 password hashing, which is expected and intentional for security. - Most other endpoints respond under 100ms, which is healthy.
Action: Investigate whether the driver management endpoints can be optimized with batch database operations or caching.
Knowing which endpoints receive the most requests tells us which features are used most frequently and where reliability matters most.
What this tells us:
-
GET /api/v1/scenarios/leads with 278 requests, meaning scenario browsing is the most-used feature. -
POST /api/v1/logs/frontendat 234 requests shows frontend logging is actively capturing user actions. - Simulation-related endpoints (
/drivers/assign,/playbackSpeed) see significant traffic (200+ requests each), confirming simulation control is a core workflow. -
GET /api/v1/users/meat 103 requests indicates frequent auth checks.
Action: Ensure the highest-traffic endpoints remain fast and reliable. These are the features users depend on most.
Tracking non-success responses (4xx and 5xx) reveals where users are encountering errors, whether from invalid input, expired sessions, or server-side bugs.
What this tells us:
-
POST /api/token(400) - 34 failed login attempts. Users entering incorrect credentials. -
POST /api/v1/simulation/{sim_id}/playbackSpeed(401) - 14 unauthorized requests. Users' auth tokens expired mid-session, causing simulation controls to fail. -
POST /api/v1/scenarios/(400) - 10 bad requests. Invalid scenario data being submitted. -
POST /api/(404) - 6 requests to a non-existent endpoint, likely a frontend bug hitting the wrong URL. -
POST /api/token(500) - 3 server errors during authentication, indicating a backend crash (possibly a database connection issue). -
GET /api/v1/users/me(500) - 1 server error, possibly a database issue.
Action: The 401 errors on playback speed suggest the frontend should proactively refresh tokens before they expire. The 500 errors on /api/token need investigation as they indicate server-side crashes during login.
Frontend action tracking shows which UI features operators interact with most, providing insight into user behavior and workflow patterns.
What this tells us:
-
user_logout(46) is the most logged action, indicating users are actively ending sessions. -
scenario_import_button(29) andscenario_save_imported(21) show the import workflow is heavily used. -
scenario_save_as_new(20) andscenario_save_new(11) show users frequently create and save scenarios. -
unsaved_scenario_changes_discarded(15) suggests users sometimes abandon changes, which could indicate UX friction. -
WebSocket connection error(5) reveals occasional real-time communication failures during simulation playback.
Action: We’ve been able to observe that roughly 65% of navigation away from an unsaved scenario was intentional, which means this unsaved changes dialog has been mostly serving as an extra obstacle for navigating within the scenario editor.
For this reason, we’re discussing options to reduce or remove the usage of this dialog, such as an auto-save feature.
Over the monitoring period:
- 198 total frontend log entries were captured
- 0 warnings and 5 errors recorded, giving a low error rate (~2.5%)
- 36 unique users interacted with the application
The low error count relative to total activity indicates the frontend is generally stable, but the 5 errors (WebSocket connection failures) should be addressed.
This chart shows when frontend activity occurs and at what severity:
- ERROR logs (green) appear in small numbers near the end of the activity spike, suggesting issues occur under load.
- The pattern shows usage is concentrated in specific time windows rather than spread evenly, which is expected for a simulation tool used during work sessions.
Action: Correlate the error spikes with backend metrics to determine if frontend errors are caused by backend performance degradation under load.
CPU and memory metrics show how close we are to the server's limits under normal usage.
What this tells us:
- CPU spikes to 100% around Jan 27-28, correlating with periods of active simulation usage. The average is 5.5% but peaks hit 101%, meaning the server is maxing out during heavy operations (likely simulation initialization or frame processing).
- Memory jumps from ~130 MiB to 150 MiB (RAM) and virtual memory climbs from ~1 GiB to 2.1 GiB during the same active period. The step-wise increases in virtual memory suggest new simulation processes or large data allocations.
- Memory Growth Rate shows small spikes but returns to 0 B/s, meaning memory is allocated during operations but not leaking. The mean of 292 B/s is negligible.
- Current state (gauges) shows 0.509% CPU and 130 MiB RAM -- the server has recovered to idle after the load period.
Action: The CPU hitting 100% is the main concern. Investigate which operations cause the spikes (likely simulation frame processing or GraphHopper routing calls). Consider whether these can be optimized or if the server needs more CPU resources for concurrent simulations.