v0.2.3 — parallel orchestrator + per-class confidence thresholds

Latency + UX release. Two independent wins that ship together:

1. Parallel Executors in the multi-agent orchestrator

The orchestrator now runs the Planner-generated subtasks concurrently within an attempt, capped by asyncio.Semaphore(MAX_CONCURRENT_EXECUTORS) (default 5, env-configurable).

Why this matters. In v0.2.2 the production smoke test captured a complex_task taking 31.7 s warm (6 subtasks × ~5 s each, sequentially) and another timing out at 60 s on cold start. The subtasks generated by the Planner are typically independent — sequential execution was leaving 5×-6× of latency on the table.

Measured against the same prompts on v0.2.3:

Prompt	v0.2.2 (sequential)	v0.2.3 (parallel)	Speedup
"Design a scalable architecture for a food delivery app…" (5 subtasks, cold start)	timeout @ 60 s	26.4 s	—
"Help me migrate a monolith to microservices step by step." (6 subtasks, warm)	31.7 s	9.2 s	~3.4×

How

• Executor.execute_async() wraps the existing sync execute() in asyncio.to_thread. Chosen over AsyncOpenAI / AsyncAnthropic because the LLMProvider ABC is sync (used by Planner/Executor/Critic/LLMRouter/EmbedRouter); switching to async would fork the entire provider hierarchy. The GIL releases during the SDK's underlying httpx socket I/O, so threads actually run concurrently for I/O-bound LLM roundtrips.
• Orchestrator.run_task_async() does asyncio.gather(..., return_exceptions=True) so a single subtask failing doesn't tear down the rest — it gets an inline [execution failed: <Type>: <msg>] marker in the aggregated answer and a EXEC[i] FAILED ... line in the trace.
• The public Orchestrator.run_task(task) keeps its sync signature for the existing FastAPI dispatcher; it now internally asyncio.run(run_task_async(task)). No changes required in dispatch.py / api.py.

Configuration

• MAX_CONCURRENT_EXECUTORS env var (default 5). Invalid/missing values fall back to the default; logged at WARNING.
• The Cloud Run deploy sets MAX_CONCURRENT_EXECUTORS=5 explicitly so the value is visible in gcloud run services describe.

2. Per-class confidence thresholds

The single global CONFIDENCE_THRESHOLD=0.65 from v0.2.2 was demoting legitimate chitchat to the fallback path: "Good morning! Hope you're having a nice day." scored 0.560 (the natural ceiling for chitchat in this model) and got rejected as ambiguous — captured in scripts/results/prod_routing_report.md.

New per-class defaults (calibrated from production data, see the README evaluation section):

Class	Threshold	Why
simple_qa	0.65	Clear inputs 0.85–0.86
complex_task	0.65	Clear inputs 0.82–0.94
document_qa	0.65	Clear inputs 0.81–0.88
chitchat	0.45	Clear inputs cap at 0.52–0.56 — naturally lower distribution

New env vars:

• CONFIDENCE_THRESHOLDS (preferred) — JSON map, e.g. '{"chitchat": 0.45, "simple_qa": 0.65}'.
• CONFIDENCE_THRESHOLD (legacy scalar) — still works as a uniform fallback for classes not specified in the JSON map.

Malformed JSON, unknown intent keys, non-numeric values, out-of-range values → logged at WARNING and ignored / clamped. The service never crashes on a misconfigured env var.

Production smoke test (same script, same inputs, run on v0.2.3)

ambiguous → fallback or conf < 0.65: 4/4 PASS
clear     → NOT fallback:            7/7 PASS   (was 5/6 FAIL in v0.2.2)
fallbacks observed:                  3/11

latency mean: 3623 ms · warm-only mean: 1344 ms  (cold-start first request: 26.4 s)

scripts/results/prod_routing_report.md regenerated and committed.

Test counts

98 tests total (was 92 in v0.2.2). New cases:

• executors_run_concurrently_and_preserve_subtask_order
• one_subtask_failure_does_not_lose_the_others
• semaphore_caps_simultaneous_executor_calls
• env_max_concurrent_overrides_default + _invalid_value_uses_default
• run_task_async_is_directly_awaitable

Plus the 10 per-class threshold tests from the same release branch.

All tests use a thread-safe FakeProvider with a content-dispatched responder so they're deterministic regardless of the order asyncio threads fire — no flakiness in 5 consecutive runs of the semaphore + concurrency tests.

Live

	URL
Backend (Cloud Run)	https://agent-router-909428365094.us-central1.run.app
Frontend (Vercel)	https://agent-router-five.vercel.app

Cloud Run deploy

gcloud run deploy agent-router \
  --image us-central1-docker.pkg.dev/research-agent-498415/agent-router/agent-router:v0.2.3 \
  --service-account=agent-router-runtime@research-agent-498415.iam.gserviceaccount.com \
  --set-env-vars='^@^...@MAX_CONCURRENT_EXECUTORS=5' \
  ...

Public API

Unchanged. POST /route returns the same RouteResponse schema. Same path_taken values plus the existing low_confidence_fallback. Same error codes (422 / 429 / 503 / 500 / 413).