Linear: [AI-2879](https://linear.app/stream/issue/AI-2879), router
launch metrics (under AI-2588).
**Ready for review. Not merged: Kanat merges after review.**
## What it does
The router's `/metrics` now serves the signals that the launch alerts
need. Every label is bounded: no customer, app, session, call or key
value.
```
/metrics
├── API requests count and latency per route template #1786 AI-2778
├── Sessions active, ends by reason / fault / stage #1782 AI-2779, #1796 AI-2831, #1839 AI-2949
├── Failover LLM / STT / TTS vendor moves per kind #1785 AI-2781
├── Turns turn latency, start-up stages and failures #1780 AI-2826, #1794 AI-2832, #1841 AI-2969
├── Silent calls caller sent no audio, per call kind #1792 AI-2828
├── Phone call outcomes per vendor and direction, #1783 AI-2780
│ carrier KPIs #1793 AI-2830
└── Rate limits vendor 429 refusals per key and key headroom #1781 AI-2833
```
A failed model call now says why, and no secret leaks on the way:
```
provider error ──► model_call frame error_message (start of the vendor's text) #1834 AI-2954
failure log line provider + model
│
└─► one redactor (store.RedactLogText / SafeLogText) #1836 AI-2958 AI-2959
├── client error frames and socket close reasons
├── every binary's slog handler (ReplaceAttr) #1838 AI-2966
├── Sentry events, contexts and tags #1838, #1846 AI-2975
└── stored tool, recording and simulation errors #1846 AI-2975
```
A lost voice no longer loses the answer:
```
TTS vendor refuses mid-turn ──► next voice says again what the caller did not hear #1840 AI-2970
held back- check-in kept, dropped if the caller talks #1843 AI-2973, #1850 AI-2977
lost voice closes in the background (no 2 s wait) #1844 AI-2974
```
Staging fixes (umbrella AI-2929):
| PR | Ticket | Fix |
|---|---|---|
| #1805 | AI-2930 | the rate-limit refusal metric is served at 0 for
each pooled key before any 429 |
| #1811 | AI-2939 | a failed reply marks the turn lost, even when the
flow-controller call succeeds |
| #1810 | AI-2940 | a table vendor with no key is served one
`key="none"` series at 0 |
| #1839 | AI-2949 | a create on an app with Video off counts as
`customer_config`, not the router's fault |
| #1840 | AI-2970 | after a TTS failover the next voice says the kept
answer again |
| #1841 | AI-2969 | turn latency counts delegated and tool answers;
scheduled turns are left out |
| #1843 | AI-2973 | a `back-` check-in is held across a voice loss |
| #1844 | AI-2974 | the lost voice closes in the background, so the next
voice starts at once |
| #1846 | AI-2975 | Sentry contexts and tags, and stored tool errors, go
through the redactor |
| #1850 | AI-2977 | test: the agent's own echo does not drop a held
check-in |
## Acceptance criteria (staging, last window on feature tip `7614abd2d`)
| AC | What | Result |
|---|---|---|
| AC-1 | request count and duration per route template, no config id in
labels | pass |
| AC-2 | active sessions go up and back; one session end with its
documented reason | pass |
| AC-3 | a refused customer LLM key ends with fault `customer_config`,
not a vendor failover | pass (after #1811) |
| AC-4 | a bad first LLM candidate counts one failover; a working one
counts none | pass (local router) |
| AC-5 | 3 turns add 3 turn-latency observations; each start-up stage
adds 1 | pass (rechecked on `7614abd2d`) |
| AC-6 | 60 s of caller silence counts one silent web call | pass
(rechecked on `7614abd2d`) |
| AC-7 | phone outcomes and carrier KPIs on an answered and an
unanswered call | **not checked**: no staging number on the test app,
and phonecheck runs only in the main session |
| AC-8 | header vendor headroom gauge, plus a refusal series for each
vendor in the table | pass (after #1805, #1810) |
| AC-9 | every existing `router_*` name is still served; no id-like
labels | pass |
| AC-10 | a refused customer OpenAI key: `model_call` frame
`success:false`, `error_message` holds the start of OpenAI's text, the
key is in no frame or log | pass (rechecked on `7614abd2d`) |
| AC-11 | a working model: `model_call` frame `success:true`, no
`error_message` key | pass |
| AC-12 | an upstream error that echoes `Authorization: Bearer …`:
`error_message` keeps the text, not the token | pass (local router) |
| AC-13 | the same failure: the router log line names provider and model
| pass (local router) |
Last window, also checked live:
- **Voice loss (AI-2974):** the time from the lost voice to the next
voice's first audio went from 2,157 ms to 120–158 ms (3 runs).
- **Check-ins:** a held check-in is said once on the next voice
(AI-2973).
- **Sentry:** no new issue, and no `sk-` or `Bearer` text in new events.
Evidence for each AC (commands, metric deltas, timestamps) is in the
verifier's comments on AI-2879.
## Gates for the merge into accelerate
None needs a flag. Migrations: none.
1. **Session API (behaviour change, AI-2831).**
- `Session` gains `end_fault` and `end_stage`.
- `SessionEndReason` gains `closed`, `shutdown`, `agent_failed`,
`model_failed`, `model_config_refused` and `model_no_credit`.
- Sessions stopped through the API, ended by a shutdown, or failed now
carry an `end_reason`.
- Python and Rust are open enums since #1828 (AI-2951). Neither package
is on a registry. Only consumers pinned to an older git commit fail.
2. **Stored values.**
- `agent_sessions.end_reason` (TEXT) now holds the 6 new values. There
is no migration.
- A router rollback is safe on the server. It does not rewrite those
rows.
3. **Redis.**
- A refused arriving call writes one SET with an expiry.
- Every inbound leg end reads one EXISTS, with a 1 s timeout.
4. **`model_call.error_message`** on the frame and on the call
timeline's `model_calls`: an optional field that only adds. No client
release.
5. **New end reasons** `create_no_video`, `create_no_call_type` and
`create_no_channel_type`: these are metric labels only. Video-off
creates move from the router's fault to `customer_config`.
6. **Redacted errors.**
- Client-visible error frames, socket close reasons and stored tool
errors now go through the redactor.
- A text over 8 KiB is cut to 2,048 runes with `… [truncated]`.
- Rows already stored are not rewritten.
7. **CLI output.** `cmd/agent`, `chat`, `say` and `knowledge` now print
their final error as a logfmt `level=ERROR` line. The exit code is still
1.
8. **Re-said text after a TTS failover.**
- The next voice says again a reply, greeting, check-in, update or hold
line the caller did not hear (#1840, #1843). This adds TTS billing for
that text.
- The lost voice closes in the background (#1844), and stopping the
agent still waits for it.
9. **Turn event timing.** The session socket `turn` event of a turn that
only delegated or called a tool now comes at the first line said for it,
with a non-zero `roundtrip_ms` (#1841). A turn that a scheduled task
starts is no longer observed.
## Series
Upper bounds per router process:
| Area | Series |
|---|---|
| HTTP | about 20.2k, only for routes that were called |
| Sessions, silent calls, start-up, turn latency, failovers, phone |
1,155 together |
| Rate limits | one per pooled key, up to 11 `key="none"`, and headroom
of at most 5 per OpenAI key, 4 per Anthropic key and 1 per ElevenLabs
key |
Measured on staging: 338 and 428 new series per pod.
## Rollout
1. Merge into accelerate. Deploy to staging, then prod as usual.
2. The Datadog monitors and dashboards built on these metrics are
separate work (AI-2588). The provider balance exporter is live
separately (AI-2953, GetStream/chat).
## Open
- AC-7 needs a phone check on staging.
- Staging's ElevenLabs account refuses library voices through the API,
so the voice ACs ran on Cartesia. Every voice-loss check used that
refusal as its failover.
- Voice ACs ran on a test app with Video on. Rerun them on the main text
test app once its Video is on.
- Follow-up tickets: AI-2943, AI-2944, AI-2950, AI-2971, AI-2976.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kanat <kanat.kiialbaev@getstream.io>
