feat(inference): enhance schemas and models for TTS and STT

[russellmartin-livekit]

Description

Brings the @livekit/agents Inference clients (LLM / STT / TTS under agents/src/inference) back in sync with the inference-gateway capabilities. This covers model-catalog drift plus three functional gaps where the gateway supports a feature the JS client never exercised: LLM response_format, STT mid-stream session.update, and TTS per-utterance generation_config.

Changes Made

• LLM (llm.ts) — Synced the model catalog with the gateway config (added zai/glm-5.1, moonshotai/kimi-k2.5/k2.6, openai/gpt-5.5/gpt-5.4-mini/gpt-5.4-nano/chat-latest, google/gemini-3.1-pro/gemini-3.1-flash-lite/gemini-3.5-flash; dropped stale entries). Added a response_format passthrough field to ChatCompletionOptions so structured-output requests reach the gateway.
• STT (stt.ts) — Split Deepgram Nova vs. Flux into DeepgramModels/DeepgramFluxModels (incl. deepgram/flux-general-multi) with a dedicated DeepgramFluxOptions (eager_eot_threshold, etc.); added cartesia/ink-2(-latest), assemblyai/u3-rt-pro, and inworld/inworld-stt-1 + InworldSTTOptions. Reworked SpeechStream.updateOptions to send a mid-stream session.update over the live socket (AssemblyAI/Flux apply it without a reconnect) instead of forcing a reconnect.
• TTS (tts.ts, api_protos.ts) — Added the xai/tts-1 provider + XaiTTSOptions; expanded Cartesia (sonic-3.5/sonic-3-latest/sonic-latest), ElevenLabs (eleven_v3), Rime (coda/mistv3/mist), and Inworld (inworld-tts-1.5) catalogs. input_transcript now carries generation_config (voice/model/language) and extra (modelOptions) so mid-stream changes ride the gateway hot path; extended the Zod schema accordingly.
• Exports (index.ts) — Exported the new public types (ZAIModels, DeepgramFluxModels/DeepgramFluxOptions, InworldSTTModels/InworldSTTOptions, XaiTTSModels/XaiTTSOptions).

Pre-Review Checklist

• Build passes: tsc --noEmit clean, ESLint clean (only pre-existing tsdoc nits in untouched InworldOptions/RimeOptions comments), 122/122 inference unit tests pass
• AI-generated code reviewed: No narration comments; comments explain only non-obvious gateway-protocol intent
• Changes explained: See above
• Scope appropriate: All changes are within agents/src/inference; each is either catalog sync or a gateway feature the client was missing
• Video demo: N/A — verified directly against the production gateway instead (see Testing)

Testing

• Automated tests added/updated (if applicable) — no new tests; existing inference/*.test.ts (incl. api_protos) cover the changed surfaces
• All tests pass — 122/122 inference unit tests
• Make sure both restaurant_agent.ts and realtime_agent.ts work properly — not run; ran a direct inference smoke test against prod instead (below)

Ran a live end-to-end smoke test against the production gateway (built the package, exercised each client; STT tests feed real TTS-synthesized speech back through the recognizers): 9/9 passed.

• LLM: openai/gpt-5.5, google/gemini-3.5-flash, zai/glm-5.1 all replied; response_format: { type: 'json_object' } returned valid JSON.
• TTS: cartesia/sonic-3 + add_timestamps produced audio and 9 word-level alignments (confirms generation_config/extra → output_alignment); xai/tts-1 produced audio.
• STT: inworld/inworld-stt-1, deepgram/flux-general-multi, and assemblyai/u3-rt-pro (with a mid-stream session.update) all transcribed the phrase correctly.

Additional Notes

• deepgram/flux-general-multi only finalizes with real-time-paced audio + a trailing silence gap (turn-based end-of-turn detection); a burst feed yields only a preflight transcript. This is model behavior, not a client issue — the client parses Flux's preflight/final events correctly.
• Catalog entries are synced from agent-gateway's config.yaml; if a model is added to the client before prod deploys it, calls for that model will be rejected by the gateway until rollout.

---

[devin-ai-integration]

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.