Skip to content

v0.2.4 — observability layer (GET /stats + structured JSON logs)

Latest
Latest

Choose a tag to compare

View all tags
@raulmn00 raulmn00 released this 07 Jun 15:05
v0.2.4
29b5468

Observability layer release. Pure data API — visualization moved to the React frontend in frontend/. Backend ships no HTML.

New endpoint

GET /stats — typed JSON snapshot of in-memory metrics

The React dashboard fetches this directly. Schema is pinned by Pydantic (StatsResponse):

{
  "since": "2026-06-07T14:59:56+00:00",
  "uptime_seconds": 253.1,
  "total_requests": 9,
  "intents": {
    "<intent_name>": {
      "count": <int>,
      "share": <float>,
      "confidence": {
        "min": <float>, "max": <float>, "mean": <float>,
        "p50": <float>, "p95": <float>, "count": <int>
      } | null
    }
  },
  "fallbacks": {
    "total": <int>,
    "rate": <float>,
    "by_attempted_intent": { "<intent>": <int>, ... }
  },
  "latency_ms": {
    "<path_taken>": { "p50": <float>, "p95": <float>, "mean": <float>, "count": <int> }
  },
  "errors": { "429": <int>, "500": <int>, "503": <int> },
  "ephemeral": true,
  "note": "..."
}

Live first taste (9 requests right after v0.2.4 deploy):

intents          count  conf p50  conf p95
chitchat             3     0.48      0.54
simple_qa            2     0.86      0.87
complex_task         2     0.94      0.94
document_qa          2     0.88      0.88

latency_ms       p50      p95      count
simple_qa:direct_llm           803 ms   812 ms       2
document_qa:rag_stub            40 ms    43 ms       2
complex_task:orchestrator    16,969 ms 24,685 ms      2
low_confidence_fallback         31 ms    31 ms       1
chitchat:direct_llm          81,462 ms — (averaged with cold-start)

fallbacks        total 1   rate 11.1%   by_attempted={"chitchat": 1}
errors           {}

The chitchat row exactly reproduces what router/results/generalization_test.txt captured — confidence saturates between 0.39 and 0.55. The per-class threshold of 0.45 lets the legitimate cases through (mean 0.47) while still catching the truly ambiguous "Tell me about the requirements" (0.385) and attributing it to chitchat in the fallback breakdown.

What's instrumented

Signal Where
Volume + share per intent dispatcher records intent per request
Confidence distribution per intent bounded deque per intent, p50/p95/mean/min/max
Latency per path_taken time.perf_counter around the whole dispatch
Fallback total + rate + by attempted intent the calibration signal for per-class thresholds
HTTP error counters 429 / 500 / 503 via the exception handlers
Structured JSON log line per request stdlib JSONFormatter; secrets/inputs filtered

Honest about ephemerality

Metrics live in process memory. They reset on every Cloud Run cold start or revision change. Two layers of disclosure:

  • ephemeral: true (stable programmatic flag the React frontend can key off)
  • note (prose explanation for humans / log readers)

A TODO in metrics_collector.py marks where a to_prometheus() exporter would go for persistence. The collector's internal shape (per-intent / per-path / per-status indexed counters) already matches the natural Prometheus label dimensions; adding the exporter later is mechanical.

Removed

  • GET /dashboard and its inline HTML (~200 lines, Chart.js via CDN) — first cut shipped this for visualization, second cut moves it to the React app. Backend goes back to data-only.
  • HTMLResponse import.

A regression test test_dashboard_endpoint_is_removed asserts the /dashboard route stays 404, so a future re-introduction of inline HTML breaks loudly.

CORS

The existing allowlist (CORS_ALLOW_ORIGINS env var on Cloud Run) already covers the Vercel frontend origins + http://localhost:5173. The middleware applies globally, so /stats is automatically reachable from the React dashboard. Verified with a real preflight from the Vercel origin.

Tests

124 tests (was 98 in v0.2.3):

  • 18 new test_metrics_collector.py cases — counters, percentiles, fallback attribution by attempted intent, bounded memory via deque maxlen, error-counter wiring, full reset, thread safety (20 threads × 100 records ⇒ exactly 2000).
  • 7 new /stats cases — empty snapshot, real /route reflected, fallback attribution end-to-end, 503/500 counters via handlers, ephemeral bool flag, CORS preflight for the frontend origin, dashboard-404 regression guard.

All pass without GPU or tokens; the conftest.py autouse fixture resets the singleton between tests to prevent cross-test leakage.

Cloud Run runtime config (unchanged)

The deploy uses the same env vars as v0.2.3 plus the new /stats endpoint:

CONFIDENCE_THRESHOLDS    (defaults; per-class)
MAX_CONCURRENT_EXECUTORS=5
ENABLE_API_DOCS=false
MAX_BODY_BYTES=10000
CORS_ALLOW_ORIGINS       (Vercel + localhost)

Live: https://agent-router-909428365094.us-central1.run.app/stats

What's next

  • React dashboard view in frontend/ consuming /stats (next feature).
  • Optional to_prometheus() export when there's a Prometheus instance to scrape — collector shape is already aligned.
Assets 2
Loading