v0.2.4 — observability layer (GET /stats + structured JSON logs)

Observability layer release. Pure data API — visualization moved to the React frontend in frontend/. Backend ships no HTML.

New endpoint

GET /stats — typed JSON snapshot of in-memory metrics

The React dashboard fetches this directly. Schema is pinned by Pydantic (StatsResponse):

{
  "since": "2026-06-07T14:59:56+00:00",
  "uptime_seconds": 253.1,
  "total_requests": 9,
  "intents": {
    "<intent_name>": {
      "count": <int>,
      "share": <float>,
      "confidence": {
        "min": <float>, "max": <float>, "mean": <float>,
        "p50": <float>, "p95": <float>, "count": <int>
      } | null
    }
  },
  "fallbacks": {
    "total": <int>,
    "rate": <float>,
    "by_attempted_intent": { "<intent>": <int>, ... }
  },
  "latency_ms": {
    "<path_taken>": { "p50": <float>, "p95": <float>, "mean": <float>, "count": <int> }
  },
  "errors": { "429": <int>, "500": <int>, "503": <int> },
  "ephemeral": true,
  "note": "..."
}

Live first taste (9 requests right after v0.2.4 deploy):

intents          count  conf p50  conf p95
chitchat             3     0.48      0.54
simple_qa            2     0.86      0.87
complex_task         2     0.94      0.94
document_qa          2     0.88      0.88

latency_ms       p50      p95      count
simple_qa:direct_llm           803 ms   812 ms       2
document_qa:rag_stub            40 ms    43 ms       2
complex_task:orchestrator    16,969 ms 24,685 ms      2
low_confidence_fallback         31 ms    31 ms       1
chitchat:direct_llm          81,462 ms — (averaged with cold-start)

fallbacks        total 1   rate 11.1%   by_attempted={"chitchat": 1}
errors           {}

The chitchat row exactly reproduces what router/results/generalization_test.txt captured — confidence saturates between 0.39 and 0.55. The per-class threshold of 0.45 lets the legitimate cases through (mean 0.47) while still catching the truly ambiguous "Tell me about the requirements" (0.385) and attributing it to chitchat in the fallback breakdown.

What's instrumented

Signal	Where
Volume + share per intent	dispatcher records intent per request
Confidence distribution per intent	bounded deque per intent, p50/p95/mean/min/max
Latency per path_taken	time.perf_counter around the whole dispatch
Fallback total + rate + by attempted intent	the calibration signal for per-class thresholds
HTTP error counters	429 / 500 / 503 via the exception handlers
Structured JSON log line per request	stdlib JSONFormatter; secrets/inputs filtered

Honest about ephemerality

Metrics live in process memory. They reset on every Cloud Run cold start or revision change. Two layers of disclosure:

• ephemeral: true (stable programmatic flag the React frontend can key off)
• note (prose explanation for humans / log readers)

A TODO in metrics_collector.py marks where a to_prometheus() exporter would go for persistence. The collector's internal shape (per-intent / per-path / per-status indexed counters) already matches the natural Prometheus label dimensions; adding the exporter later is mechanical.

Removed

• GET /dashboard and its inline HTML (~200 lines, Chart.js via CDN) — first cut shipped this for visualization, second cut moves it to the React app. Backend goes back to data-only.
• HTMLResponse import.

A regression test test_dashboard_endpoint_is_removed asserts the /dashboard route stays 404, so a future re-introduction of inline HTML breaks loudly.

CORS

The existing allowlist (CORS_ALLOW_ORIGINS env var on Cloud Run) already covers the Vercel frontend origins + http://localhost:5173. The middleware applies globally, so /stats is automatically reachable from the React dashboard. Verified with a real preflight from the Vercel origin.

Tests

124 tests (was 98 in v0.2.3):

• 18 new test_metrics_collector.py cases — counters, percentiles, fallback attribution by attempted intent, bounded memory via deque maxlen, error-counter wiring, full reset, thread safety (20 threads × 100 records ⇒ exactly 2000).
• 7 new /stats cases — empty snapshot, real /route reflected, fallback attribution end-to-end, 503/500 counters via handlers, ephemeral bool flag, CORS preflight for the frontend origin, dashboard-404 regression guard.

All pass without GPU or tokens; the conftest.py autouse fixture resets the singleton between tests to prevent cross-test leakage.

Cloud Run runtime config (unchanged)

The deploy uses the same env vars as v0.2.3 plus the new /stats endpoint:

CONFIDENCE_THRESHOLDS    (defaults; per-class)
MAX_CONCURRENT_EXECUTORS=5
ENABLE_API_DOCS=false
MAX_BODY_BYTES=10000
CORS_ALLOW_ORIGINS       (Vercel + localhost)

Live: https://agent-router-909428365094.us-central1.run.app/stats

What's next

• React dashboard view in frontend/ consuming /stats (next feature).
• Optional to_prometheus() export when there's a Prometheus instance to scrape — collector shape is already aligned.

v0.2.3 — parallel orchestrator + per-class confidence thresholds

Latency + UX release. Two independent wins that ship together:

1. Parallel Executors in the multi-agent orchestrator

The orchestrator now runs the Planner-generated subtasks concurrently within an attempt, capped by asyncio.Semaphore(MAX_CONCURRENT_EXECUTORS) (default 5, env-configurable).

Why this matters. In v0.2.2 the production smoke test captured a complex_task taking 31.7 s warm (6 subtasks × ~5 s each, sequentially) and another timing out at 60 s on cold start. The subtasks generated by the Planner are typically independent — sequential execution was leaving 5×-6× of latency on the table.

Measured against the same prompts on v0.2.3:

Prompt	v0.2.2 (sequential)	v0.2.3 (parallel)	Speedup
"Design a scalable architecture for a food delivery app…" (5 subtasks, cold start)	timeout @ 60 s	26.4 s	—
"Help me migrate a monolith to microservices step by step." (6 subtasks, warm)	31.7 s	9.2 s	~3.4×

How

• Executor.execute_async() wraps the existing sync execute() in asyncio.to_thread. Chosen over AsyncOpenAI / AsyncAnthropic because the LLMProvider ABC is sync (used by Planner/Executor/Critic/LLMRouter/EmbedRouter); switching to async would fork the entire provider hierarchy. The GIL releases during the SDK's underlying httpx socket I/O, so threads actually run concurrently for I/O-bound LLM roundtrips.
• Orchestrator.run_task_async() does asyncio.gather(..., return_exceptions=True) so a single subtask failing doesn't tear down the rest — it gets an inline [execution failed: <Type>: <msg>] marker in the aggregated answer and a EXEC[i] FAILED ... line in the trace.
• The public Orchestrator.run_task(task) keeps its sync signature for the existing FastAPI dispatcher; it now internally asyncio.run(run_task_async(task)). No changes required in dispatch.py / api.py.

Configuration

• MAX_CONCURRENT_EXECUTORS env var (default 5). Invalid/missing values fall back to the default; logged at WARNING.
• The Cloud Run deploy sets MAX_CONCURRENT_EXECUTORS=5 explicitly so the value is visible in gcloud run services describe.

2. Per-class confidence thresholds

The single global CONFIDENCE_THRESHOLD=0.65 from v0.2.2 was demoting legitimate chitchat to the fallback path: "Good morning! Hope you're having a nice day." scored 0.560 (the natural ceiling for chitchat in this model) and got rejected as ambiguous — captured in scripts/results/prod_routing_report.md.

New per-class defaults (calibrated from production data, see the README evaluation section):

Class	Threshold	Why
simple_qa	0.65	Clear inputs 0.85–0.86
complex_task	0.65	Clear inputs 0.82–0.94
document_qa	0.65	Clear inputs 0.81–0.88
chitchat	0.45	Clear inputs cap at 0.52–0.56 — naturally lower distribution

New env vars:

• CONFIDENCE_THRESHOLDS (preferred) — JSON map, e.g. '{"chitchat": 0.45, "simple_qa": 0.65}'.
• CONFIDENCE_THRESHOLD (legacy scalar) — still works as a uniform fallback for classes not specified in the JSON map.

Malformed JSON, unknown intent keys, non-numeric values, out-of-range values → logged at WARNING and ignored / clamped. The service never crashes on a misconfigured env var.

Production smoke test (same script, same inputs, run on v0.2.3)

ambiguous → fallback or conf < 0.65: 4/4 PASS
clear     → NOT fallback:            7/7 PASS   (was 5/6 FAIL in v0.2.2)
fallbacks observed:                  3/11

latency mean: 3623 ms · warm-only mean: 1344 ms  (cold-start first request: 26.4 s)

scripts/results/prod_routing_report.md regenerated and committed.

Test counts

98 tests total (was 92 in v0.2.2). New cases:

• executors_run_concurrently_and_preserve_subtask_order
• one_subtask_failure_does_not_lose_the_others
• semaphore_caps_simultaneous_executor_calls
• env_max_concurrent_overrides_default + _invalid_value_uses_default
• run_task_async_is_directly_awaitable

Plus the 10 per-class threshold tests from the same release branch.

All tests use a thread-safe FakeProvider with a content-dispatched responder so they're deterministic regardless of the order asyncio threads fire — no flakiness in 5 consecutive runs of the semaphore + concurrency tests.

Live

	URL
Backend (Cloud Run)	https://agent-router-909428365094.us-central1.run.app
Frontend (Vercel)	https://agent-router-five.vercel.app

Cloud Run deploy

gcloud run deploy agent-router \
  --image us-central1-docker.pkg.dev/research-agent-498415/agent-router/agent-router:v0.2.3 \
  --service-account=agent-router-runtime@research-agent-498415.iam.gserviceaccount.com \
  --set-env-vars='^@^...@MAX_CONCURRENT_EXECUTORS=5' \
  ...

Public API

Unchanged. POST /route returns the same RouteResponse schema. Same path_taken values plus the existing low_confidence_fallback. Same error codes (422 / 429 / 503 / 500 / 413).

v0.2.2 — confidence threshold + low-confidence fallback

Confidence threshold + low-confidence fallback in the routing dispatcher. Backward compatible at the API surface — RouteResponse schema is unchanged; only a new value for path_taken is possible.

What's new

CONFIDENCE_THRESHOLD (env, default 0.65)

When IntentClassifier.classify() returns a confidence below the threshold, the dispatcher skips the four normal routing arms and lands on a transparent fallback that:

• Reports the model's best-guess intent and the actual (low) confidence.
• Sets path_taken = "low_confidence_fallback".
• Puts the would-have-been path into the trace, so the operator can see what the router would have done.
• Returns a user-facing message explaining the uncertainty — no fake answer.
• Makes no LLM call. The fallback is intentionally cheap; an LLM-driven clarifying-question step is the next natural extension.

$ curl -X POST $URL/route -d '{"input":"Tell me about the requirements"}'
{
  "intent": "chitchat",
  "confidence": 0.493,
  "path_taken": "low_confidence_fallback",
  "answer": "I couldn't confidently classify your request — the most likely intent was 'chitchat' with confidence 0.493, below the threshold of 0.65. Try rephrasing with a clearer cue.",
  "trace": [
    "low confidence: intent='chitchat' confidence=0.493 threshold=0.65",
    "would have routed to: chitchat:direct_llm",
    "no LLM call made; future: ask clarifying question or escalate to LLM router"
  ]
}

Empirical motivation

The 0.65 default isn't arbitrary. The generalization probe committed at router/results/generalization_test.txt (20 fresh sentences, never seen during training) shows two clean regimes:

• Clear inputs (a human would know the intent at a glance): confidence 0.81–0.94.
• Ambiguous inputs (deliberately stripped of class markers): confidence 0.39–0.59.

0.65 falls cleanly in the gap and produces no false negatives on the clear set and no false positives on the ambiguous set.

Defensive parsing

• CONFIDENCE_THRESHOLD parses as float; on ValueError falls back to the default rather than crashing the service.
• Values outside [0, 1] are clamped — a typo like 9.0 can't accidentally turn off the gate.

Tests

Eight new test cases in backend/tests/test_api.py:

• test_high_confidence_routes_through_normal_path
• test_low_confidence_falls_back_without_calling_llm — asserts the provider factory is never called in the fallback (no token spend)
• test_low_confidence_trace_includes_attempted_intent_and_confidence
• test_low_confidence_answer_does_not_pretend_to_have_classified
• test_explicit_threshold_overrides_default
• test_env_var_configures_threshold
• test_env_var_invalid_value_falls_back_to_default
• test_env_var_out_of_range_is_clamped

82 backend tests total now (was 74).

Live verification

URL=https://agent-router-909428365094.us-central1.run.app

# High confidence — normal route
curl -X POST $URL/route -H 'Content-Type: application/json' \
  -d '{"input":"What is the capital of France?"}' | jq .path_taken
# "simple_qa:direct_llm"

# Low confidence — fallback path, no LLM call (~300ms warm)
curl -X POST $URL/route -H 'Content-Type: application/json' \
  -d '{"input":"Tell me about the requirements"}' | jq .path_taken
# "low_confidence_fallback"

Cloud Run deploy

gcloud run deploy agent-router \
  --image us-central1-docker.pkg.dev/research-agent-498415/agent-router/agent-router:v0.2.2 \
  --service-account=agent-router-runtime@research-agent-498415.iam.gserviceaccount.com \
  --set-env-vars='^@^...@CONFIDENCE_THRESHOLD=0.65' \
  ...

Not in this release

• The fallback does not call any LLM to ask a clarifying question. That's the next natural extension and is left as a marker in the code (Dispatcher._low_confidence_fallback docstring).
• The trained DistilBERT model used by the live service is still the v0.1.0 GitHub Release artifact (MPS-trained on the local laptop). The Colab-trained CUDA model (the one that produced the perfect confusion matrix in router/results/) hasn't been published as a release yet — confidence values in production may differ slightly from the numbers in the README's generalization probe until the model in the release is updated.

v0.2.1 — security hardening

Security hardening release. No behavior changes to /route or /compare for legitimate clients; all changes are defense-in-depth.

Changes

Cloud Run — dedicated minimum-privilege SA

The runtime SA is now agent-router-runtime@research-agent-498415.iam.gserviceaccount.com with only roles/secretmanager.secretAccessor on the openai-api-key secret. The previous default Compute Engine SA had project-wide roles/editor — if the container had been compromised, an attacker could read/modify every Secret Manager secret, modify other Cloud Run services, etc.

Sanitized /compare per-row error messages

backend/app/compare.py no longer echoes f"{type(e).__name__}: {e}" to the client. Errors collapse to a fixed allowlist:

• FileNotFoundError → "router artifact not available"
• ProviderUnavailableError → "upstream provider not configured"
• RateLimitError → "upstream provider rate limited"
• APIConnectionError → "upstream provider temporarily unreachable"
• (etc., with "router unavailable" as the fallback)

Full exceptions are still logged server-side via logger.warning for debugging.

CORS — no more wildcard with credentials

CORS_ALLOW_ORIGINS=* together with allow_credentials=True was a CORS spec violation. Now: explicit allowlist in production (https://agent-router-five.vercel.app, http://localhost:5173), and the combo * + credentials is auto-corrected (credentials disabled).

Body-size middleware

MAX_BODY_BYTES env var (default 10000) returns 413 payload too large based on Content-Length before parsing. Pydantic's max_length=2000 validation no longer pays the cost of buffering 32 MB of junk.

Security headers on every response

• X-Content-Type-Options: nosniff
• X-Frame-Options: DENY
• Referrer-Policy: no-referrer
• Strict-Transport-Security: max-age=31536000; includeSubDomains
• Cross-Origin-Resource-Policy: same-site

/docs, /redoc, /openapi.json hidden in production

Gated by ENABLE_API_DOCS (default true for local dev). The Cloud Run service sets it to false. Local dev keeps the interactive Swagger UI.

Non-root container

The image now creates a system user app (UID 1000) and switches via USER app after install. Limits the blast radius of any container escape.

Frontend: npm audit clean

Bumped vite 5 → 8 and @vitejs/plugin-react 4.3 → 6.0. The previous moderate-severity advisory (GHSA-67mh-4wv8-2f99, esbuild dev server) is resolved; npm audit reports 0 vulnerabilities. Production bundle size is unchanged.

New tests

• test_security_headers_present_on_every_response
• test_body_size_limit_returns_413
• test_body_size_limit_with_malformed_content_length_returns_400
• test_compare_one_router_failure_does_not_break_response (updated to pin the sanitized error string)

22 backend tests total (was 19). 65 across the workspace.

Live

	URL
Frontend	https://agent-router-five.vercel.app
Backend	https://agent-router-909428365094.us-central1.run.app

Verifying the live hardening

URL=https://agent-router-909428365094.us-central1.run.app

# /docs hidden
curl -sI $URL/docs            # → HTTP/2 404
curl -sI $URL/openapi.json    # → HTTP/2 404

# Security headers present
curl -sI $URL/ | grep -iE 'nosniff|strict-transport|x-frame|referrer|cross-origin'

# CORS allows Vercel
curl -sI -X OPTIONS $URL/route \
  -H 'Origin: https://agent-router-five.vercel.app' \
  -H 'Access-Control-Request-Method: POST' | grep -i access-control-allow-origin

# CORS blocks unknown origin (no Allow-Origin in response)
curl -sI -X OPTIONS $URL/route \
  -H 'Origin: https://evil.example.com' \
  -H 'Access-Control-Request-Method: POST' | grep -i access-control-allow-origin || echo "blocked"

# 413 on oversized body
python3 -c 'import json; print(json.dumps({"input":"x"*15000}))' \
  | curl -s -X POST $URL/route -H 'Content-Type: application/json' --data-binary @- -w '\nHTTP %{http_code}\n'

v0.2.0 — /compare endpoint + React frontend + rate limiting

Demo layer release: a side-by-side comparison endpoint and a production React frontend that consumes it.

Live

	URL
Frontend (Vercel)	https://agent-router-five.vercel.app
Backend (Cloud Run)	https://agent-router-909428365094.us-central1.run.app
Model release	https://github.com/raulmn00/agent-router/releases/tag/v0.1.0

Highlights

• POST /compare endpoint — runs all three routers (DistilBERT, LLM zero-shot via gpt-4o-mini, embeddings + LogReg) on the same input and returns one row per router with measured latency_ms, derived confidence, computed cost_per_1k_usd, and either an intent or an error string. Each router is isolated behind try/except — a single failure leaves the others' rows intact and the endpoint still returns 200.
• React + TypeScript frontend (Vite, hand-written CSS, no UI library, strict: true, no any). Two modes: Roteamento (single /route call → IntentBadge + ConfidenceGauge + PathFlow + collapsible Trace; the Trace auto-expands when intent is complex_task to show Planner → Executors → Critic) and Comparação (three columns revealing in the order of the measured latency).
• Confidence everywhere — LLMRouter.classify_with_confidence derives a [0, 1] score from the OpenAI logprobs of the first output token; EmbedRouter.classify_with_confidence returns predict_proba.max(). OpenAIProvider.complete_with_logprobs is a new optional method on the provider abstraction.
• Rate limiting via slowapi — 30 req/min/IP on /route, 10 req/min/IP on /compare (it burns real tokens). 429 returns a friendly message; the frontend handles it gracefully.
• Persisted embeddings LogReg — python -m eval.fit_embed_router saves the fitted classifier to eval/models/embed_router.joblib (~25 KB, committed). The backend loads it once at startup; cold-start no longer pays the training cost.
• ProviderUnavailableError — typed exception in agents/providers.py mapped to 503 by the backend with a generic message that never leaks the env var name.

Migration from v0.1.0

• New backend dependency: slowapi==0.1.* (in backend/requirements.txt).
• Optional env var: EMBED_ROUTER_PATH (defaults to <repo>/eval/models/embed_router.joblib).
• The Dockerfile now copies eval/ into the image and adds it to PYTHONPATH. Cloud Run deploys must be re-built — the v0.1.0 image will start but /compare would 500 with ModuleNotFoundError: eval.
• /route is behavior-compatible: same request, same response, same 422/503/500 error codes. The only addition is 429 on overuse.
• Frontend is brand-new (frontend/ subdir of the monorepo). Deployed independently on Vercel; VITE_API_URL points at the Cloud Run backend.

Test counts

router/tests/    11  schema, balance, stratified split, classifier softmax/argmax
agents/tests/    15  orchestrator happy path + critic-reproves-and-retries
backend/tests/   19  /route 4 dispatch paths + /compare cases + 422/429/503/500
eval/tests/      17  accuracy, f1_macro, cost_per_1k pinning + sklearn cross-check
                 ──
                 62

npm run build in frontend/ runs tsc -b first — type-check failures break the build.

v0.1.0 — initial fine-tuned router model

DistilBERT-base-uncased fine-tuned on 600 synthetic examples (150 per class), 4 epochs.

Held-out evaluation (40 handcrafted examples, see eval/data/routing_testset.jsonl):

• Accuracy: 0.975
• F1 (macro): 0.975

The Cloud Run / Docker entrypoint pulls this tarball at container start when MODEL_RELEASE_URL points to it.