[bug] model call latency histogram buckets top out at 10s, so p99 is silently capped

[yanyanz89]

Symptom

switchyard_model_call_latency_ms and switchyard_total_latency_ms use the OpenTelemetry SDK's default explicit bucket boundaries, whose largest finite bucket is 10000 ms. Any model call slower than 10s falls into +Inf, and histogram_quantile can then only return 10000. A p99 of 10s, 40s, and 5 minutes are indistinguishable on /metrics.

Reproduction

./target/release/switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"switchyard/split","messages":[{"role":"user","content":"say hi"}],"max_tokens":48}'

curl -s http://127.0.0.1:4000/metrics | grep model_call_latency_ms_bucket

Observed boundaries:

​ le="0" le="5" le="10" le="25" le="50" le="75" le="100" le="250" le="500" le="750" le="1000" le="2500" le="5000" le="7500" le="10000" le="+Inf" ​

switchyard_routing_overhead_ms is unaffected because crates/switchyard-server/src/metrics.rs installs a view for it specifically (routing_overhead_buckets, matching on instrument.name() != "switchyard.routing_overhead_ms"). Every other histogram falls through to the SDK defaults.

Expected vs. actual

• Expected: latency quantiles remain meaningful across the range LLM calls actually occupy.
• Actual: histogram_quantile(0.99, ...) clamps to exactly 10000 whenever the quantile lands in the +Inf bucket, producing a flat line at 10s that hides the real tail.

Why it matters

The default boundaries are a general-purpose set tuned for sub-second RPC. LLM serving routinely exceeds 10s on long generations, reasoning models, and large contexts, and the tail is precisely what routing decisions are evaluated on. A p99 that cannot exceed 10s makes the latency families unusable for the comparison Switchyard exists to support: is the capable tier worth its extra latency?

Alerting is affected the same way, since a threshold above 10s can never fire.

Environment

• Switchyard version (or commit SHA): switchyard_build_info{version="0.2.0"}
• Python version (python --version): Rust switchyard-server path
• OS / arch: macOS, aarch64
• Install path (uv sync, pip install nemo-switchyard, source build, etc.): source build, cargo build --release -p switchyard-server
• Inbound format (Chat Completions / Anthropic Messages / Responses): Chat Completions
• Backend (OpenAI / Anthropic / NVIDIA Inference Hub / other): NV Inference Hub