v1.7.0

Added

• Added token usage metrics to AWSNovaSonicLLMService, which now emits LLMUsageMetricsData from Nova Sonic's usageEvent. Each event reports its token delta, with speech and text tokens combined into prompt_tokens and completion_tokens, so metrics summed over a session match the session total.
  (PR #4783)

• Added an http_client parameter to OpenAITTSService and Whisper-based STT services (BaseWhisperSTTService, OpenAISTTService, GroqSTTService), so a custom httpx.AsyncClient — e.g. one with a raised request timeout for high-latency endpoints — can be used for API requests. Prefer openai.DefaultAsyncHttpxClient, which retains the OpenAI SDK's connection limits and redirect handling.
  (PR #4941)

• Added the ability for users to provide a local image to the LemonSlice transport to be used as the avatar image.
  (PR #4977)

• Added a filter_background_audio setting to ElevenLabsRealtimeSTTService.Settings so callers can have ElevenLabs suppress background and far-end audio before transcription, which reduces spurious partial transcripts and empty commits on noisy telephony audio. When set, it is forwarded as a connection query parameter regardless of commit strategy. Defaults to unset, which preserves ElevenLabs' default of no filtering.
  (PR #5003)

• Added STT usage metrics: every STT service reports usage as STTUsageMetricsData, carrying the client-measured seconds of audio submitted to the service (audio_seconds). Continuous services emit incrementally per final transcript with a flush on stop/cancel; segmented services emit per transcribed segment. Enabled with enable_usage_metrics=True; usage is forwarded to RTVI clients (stt_usage), logged by MetricsLogObserver, and attached to OpenTelemetry stt spans as metrics.audio_seconds.
  (PR #5055)

• Added inbound SIP DTMF support on the LiveKit transport via on_dtmf_event and InputDTMFFrame, so DTMFAggregator works with LiveKit SIP/PSTN calls.
  (PR #5097)

• Added numerals setting to DeepgramFluxSTTSettings to convert spoken numbers to numeral form in transcripts (e.g. "twenty three" → "23"). Enable with numerals=True in the settings; configured at connection time per Deepgram's Flux API.
  (PR #5099)

• Added safety_settings to GoogleLLMService.Settings (and GoogleVertexLLMService.Settings), exposing Gemini's content safety filters. Previously these could only be set by smuggling them through extra.

  ```python
  from google.genai.types import HarmBlockThreshold, HarmCategory,
  

  SafetySetting

  llm = GoogleLLMService(
      api_key=os.getenv("GOOGLE_API_KEY"),
      settings=GoogleLLMService.Settings(
          safety_settings=[
              SafetySetting(
                  category=HarmCategory.HARM_CATEGORY_HATE_SPEECH,
                  threshold=HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,
              ),
          ],
      ),
  )
  ```
  
  Categories left unspecified keep the Gemini API defaults. The setting is
  

  runtime-updatable via LLMUpdateSettingsFrame, which also accepts plain
  dicts.
  (PR #5109)

• Added language_codes setting to AssemblyAISTTSettings, exposing the list-valued name of AssemblyAI's declared-language parameter and the one to prefer over the singular language_code, which stays supported and is ignored when both are set. It takes Language enums: a single language (e.g. language_codes=[Language.ES]) pins transcription to that language, while several (e.g. language_codes=[Language.EN, Language.ES]) steer toward that subset while keeping code-switching among them. Steering is prompt-based, so it applies to U3 Pro models only and isn't sent forother models; on U3 Pro a change applies to the live connection instead of reconnecting, and an empty list clears steering back to the model default. At most 10 distinct languages can be declared — regional variants resolve to their base code, so they collapse rather than counting twice. An over-long list raises at construction; an over-long runtime update is dropped with a warning, leaving the current steering in place.
  (PR #5129)

• Added endpointing, keywords, and format to SmallestSTTService.Settings. endpointing finalizes transcripts promptly on trailing silence, keywords boosts recognition of domain-specific words/phrases (e.g. "Blackwell:2,NVIDIA:1"), and format controls whether transcripts getpunctuation and capitalization applied.
  (PR #5137)

• Added an opt-in force_locale setting to AzureTTSService and AzureHttpTTSService (AzureTTSSettings) that wraps synthesized text in SSML's <lang xml:lang> element, so multilingual voices (e.g. en-US-EmmaMultilingualNeural) speak in the configured locale/accent instead of auto-detecting one per segment. Defaults to False, leaving SSML output untouched unless enabled.
  (PR #5154)

• Added reach_inactive_services to ServiceUpdateSettingsFrame, and so to LLMUpdateSettingsFrame, TTSUpdateSettingsFrame and STTUpdateSettingsFrame. Set it when a settings update is provider-neutral and needs to survive a service switch, and every service a ServiceSwitcher manages applies it rather than the active one alone.

  ```python
  update = LLMUpdateSettingsFrame(
      delta=LLMSettings(temperature=0.2),
      reach_inactive_services=True,
  )
  await worker.queue_frames([update])
  ```
  
  It defaults to `False`, which suits values only one provider understands:
  

  a Cartesia voice id applied to a Deepgram TTS service would leave it unusable
  once it takes over. To configure one specific service, address the update to
  it with service= instead.
  (PR #5155)

• Added keyterm to CartesiaSTTService.Settings and CartesiaTurnsSTTService.Settings, biasing transcription toward domain-specific words and phrases such as product names and jargon.

  ```python
  stt = CartesiaTurnsSTTService(
      api_key=os.environ["CARTESIA_API_KEY"],
      settings=CartesiaTurnsSTTService.Settings(keyterm=["Pipecat", "Ink
  

  2"]),
  )
  ```

  Cartesia binds keyterms to a connection, so updating them with an
  

  STTUpdateSettingsFrame reconnects to apply them. Lists longer than
  Cartesia's limit of 100 keyterms or 1200 total characters are truncated with
  a warning. CartesiaSTTService sends keyterms only for ink-2 models, the
  only family Cartesia supports them on.
  (PR #5168)

• Added PocketTTSService, a local CPU-only text-to-speech service built on kyutai-labs' pocket-tts streaming model. Supports English, French, German, Italian, Portuguese, and Spanish, predefined voices, and voice cloning from a wav file or hf:// voice prompt. Install with pip install "pipecat-ai[pocket-tts]".

  ```python
  from pipecat.services.pocket_tts.tts import PocketTTSService
  
  tts = PocketTTSService(settings=PocketTTSService.Settings(voice="alba"))
  ```
  

  (PR #5170)

Changed

• SmallestSTTService now uses the Waves v4 STT endpoint (/waves/v1/stt/live) and sends finalize per utterance to keep the WebSocket session alive across turns.
  (PR #4747)

• The TeXML served by the development runner for Twilio and Telnyx no longer includes a trailing <Pause length="40"/>. <Connect><Stream> holds the call for the duration of the WebSocket session, so the call now hangs up as soon as the stream ends instead of lingering for another 40 seconds.
  (PR #5135)

• SageMakerBidiClient now relies on the SageMaker Runtime SDK's default auth configuration (SigV4 for the sagemaker service) instead of passing an equivalent explicit configuration, so future SDK auth changes are picked up automatically.
  (PR #5144)

• ⚠️ TavusParams.audio_out_faster_than_realtime now defaults to True. Bot audio is accumulated into 100ms chunks and sent to Tavus as fast as it is produced, giving the avatar a larger rendering buffer, instead of being paced to real playback time. Pipelines that need bot audio to arrive at roughly real time — for example when an AudioBufferProcessor downstream is recording the conversation — should now set audio_out_faster_than_realtime=False explicitly.
  (PR #5162)

• Websocket-based services now bound the websocket closing handshake at 2 seconds instead of the websockets default of 10. A service disconnects while handling the EndFrame, before the frame continues downstream, so a peer that never acknowledges the close delayed pipeline shutdown bythat much per service. The bound is configurable per service via the new ws_close_timeout argument on WebsocketService.
  (PR #5164)

• CartesiaSTTService now connects with Cartesia-Version: 2026-03-01, the version that supports keyterm and returns structured JSON errors. This brings it in line with CartesiaTTSService and `CartesiaTurnsSTTS...

v1.6.0

Added

• Added MOQTransport, a Media over QUIC (MoQ) transport that gives bots a bidirectional, low-latency audio + RTVI channel over QUIC instead of WebRTC or WebSockets. Install with pip install pipecat-ai[moq] and see examples/transports/transports-moq.py.

  • The bot runs as its own MoQ server (serve=True) and accepts the browser's direct connection, removing the need for a separate moq-relay process in local dev; client mode (dialingan external relay) is wired up but not yet enabled.
  • Audio rides a single Opus track; RTVI messages (including the transcript) ride a compressed, ordered JSON stream track, so MoQ is on par with the Daily and WebSocket transports for RTVI support.
  • The development runner (pipecat.runner.run) gained --moq-serve, --moq-bind, --moq-tls-generate/--moq-tls-cert/--moq-tls-key and related flags to configure the MoQ server and TLS for local dev.
    (PR #4629)

• Added reasoning support to OpenAIResponsesLLMService and OpenAIResponsesHttpLLMService. Set settings.reasoning to an OpenAIResponsesLLMService.ReasoningConfig(effort=..., summary=...) to control reasoning depth and, optionally, request a summary of the model's thinking. Summaries are surfaced the same way as Anthropic/Gemini thinking — as thought frames and the on_assistant_thought event. Reasoning is only supported by reasoning-capable models (the gpt-5.x series and the o-series); the default model, gpt-4.1, does not reason — see OpenAI's reasoning guide to pick a model.

  The model's encrypted reasoning is captured and sent back on subsequent
  turns automatically, preserving reasoning context across the conversation
  (and, with function calling, across tool-call turns). See
  examples/thinking/thinking-openai-responses.py (plus the -http and
  -functions- variants).

  When reasoning is not configured, the mainline gpt series from gpt-5
  onward defaults to effort="none" (reasoning disabled) to keep latency low
  for real-time voice — mirroring how the Gemini service disables thinking by
  default — while every other model is left at its provider default.
  Conversely, if you configure reasoning on a model known not to support it
  (e.g. gpt-4.1), the service logs a clear error up front instead of leaving
  you to decipher the raw API failure.
  (PR #4933)

• Added NO_RESPONSE to Pipecat Flows: a consolidated function can return (result, NO_RESPONSE) to finish the function call without transitioning to a new node or running the LLM. The next response can then be triggered by the next user utterance, or programmatically another way.
  (PR #4995)

• Added absent: true to eval scenario expectations: the expectation passes only when no event of the given type arrives within the within_ms budget, and fails as soon as one does. Useful for duplicate-output regressions, e.g. asserting a bot responds exactly once after a multi-worker handoff.
  (PR #4995)

• Added CrusoeLLMService, an OpenAI-compatible LLM service for Crusoe Cloud's Managed Inference API.
  (PR #5024)

• Added audio token usage to LLMTokenUsage for cost attribution with realtime models: optional input_audio_tokens, output_audio_tokens, and cache_read_input_audio_tokens fields. OpenAIRealtimeLLMService (and Azure realtime) now populates them from the Realtime API's response.done usage details, and they flow through the usage debug logs, RTVI client metrics (onlypresent when populated), and OTel span attributes (gen_ai.usage.audio.input_tokens, gen_ai.usage.audio.output_tokens, gen_ai.usage.audio.cache_read.input_tokens).
  (PR #5050)

• Added audio token usage capture to GeminiLiveLLMService: the AUDIO entries from usage_metadata's per-modality breakdowns now populate LLMTokenUsage's input_audio_tokens, output_audio_tokens, and cache_read_input_audio_tokens, flowing through usage logs, RTVI client metrics, and the gen_ai.usage.audio.* span attributes. Absent modalities are reported as unset rather than zero, and text tokens are never derived from totals (Gemini's modality details don't always sum to prompt_token_count). Gemini Live spans also now include cached and reasoningtoken counts, which the metrics path reported but spans were missing.
  (PR #5052)

• The development runner now prints a bordered startup banner flagging it as development-only, with a link to the deployment docs for running bots locally and in production.
  (PR #5060)

• Added DeepgramFluxTTSService, a websocket TTS service for Deepgram's Flux TTS (early access) at wss://api.deepgram.com/v2/speak. LLM tokens are streamed straight to the server as they arrive (TextAggregationMode.TOKEN, the default for this service; pass text_aggregation_mode=TextAggregationMode.SENTENCE to aggregate sentences instead) and each bot response is synthesized as a discrete turn, with prosody carried across turns on a single connection. Flux does not yet provide a way to cancel the active turn, so interruptions reconnect the websocket; examples/voice/voice-deepgram-flux.py is now an all-Flux bot (Flux STT + Flux TTS).
  (PR #5067)

• Added BasetenLLMService, an OpenAI-compatible LLM service for Baseten's Model APIs and dedicated deployments.

  Defaults to Baseten's serverless Model APIs endpoint, which serves
  

  open-weights models including GLM, Kimi, DeepSeek, Nemotron, and gpt-oss. To
  use a model running on your own dedicated GPUs, pass that deployment's
  /sync/v1 URL as base_url and set model to its served model name:

  ```python
  llm = BasetenLLMService(
      api_key=os.getenv("BASETEN_API_KEY"),
      base_url=deployment_url,
      settings=BasetenLLMService.Settings(
          model="Qwen/Qwen2.5-3B-Instruct",
      ),
  )
  ```
  

  (PR #5077)

• DailyTransport now broadcasts an STTMetadataFrame with Deepgram's TTFS P99 latency when transcription_enabled=True and transcription starts successfully, matching standalone STT services. Downstream consumers like LLMUserAggregator and the user-turn-stop strategies now use the correct STT latency instead of falling back to defaults.
  (PR #5088)

Changed

• Changed the default ElevenLabs TTS model from eleven_turbo_v2_5 to eleven_flash_v2_5 in ElevenLabsTTSService and ElevenLabsHttpTTSService, since eleven_turbo_v2_5 is now deprecated by ElevenLabs. This only affects users who don't explicitly set a model.
  (PR #4999)

• Bumped the minimum nltk version to 3.10.0.
  (PR #5019)

• ⚠️ The RTVI dtmf client message now carries buttons — a list of keypad entries, e.g. {"type": "dtmf", "data": {"buttons": ["1", "2", "#"]}} — so a single message can press a whole key sequence. The previous single-key button field is no longer accepted, and RTVI.PROTOCOL_VERSION is now 2.1.0. The RTVIProcessor pushes one InputDTMFFrame per key, in order, so downstream DTMF handling (e.g. a DTMFAggregator) behaves exactly as before.
  (PR #5030)

• Removed the pyyaml-include dependency (GPL-3.0), replacing it with a small built-in !include constructor for eval scenarios.
  (PR #5037)

• Updated tracing span attributes to the current OpenTelemetry GenAI semantic conventions: gen_ai.provider.name is now azure.ai.openai (was az.ai.openai) for AzureLLMService, x_ai (was xai) for GrokLLMService, and mistral_ai (was mistral) for MistralLLMService; reasoning token usage is now reported as gen_ai.usage.reasoning.output_tokens (was gen_ai.usage.reasoning_tokens). Update any dashboards or queries filtering on the old values.
  (PR #5047)

• Changed OpenAI Realtime llm_response span attributes to standard OTel GenAI names: tokens.prompt/tokens.completion/tokens.total are now gen_ai.usage.input_tokens/gen_ai.usage.output_tokens, plus the new cached/audio breakdown attributes. Update any dashboards or queries filtering on the old tokens.* names.
  (PR #5050)

• Changed Gemini Live llm_response span attributes: the non-standard tokens.prompt/tokens.completion/tokens.total were removed in favor of the standard gen_ai.usage.input_tokens/gen_ai.usage.output_tokens attributes already present on the same spans. Update any dashboards or queries filtering on the old tokens.* names.
  (PR #5052)

• Updated the runner extra to require pipecat-ai-prebuilt>=1.0.4.
  (PR #5061)

• Updated the runner extra to require pipecat-ai-prebuilt>=1.0.5 to add support for the MoQ transport.
  (PR #5073)

• TTSService now logs Generating TTS [text] itself, just before invoking run_tts: at debug level in sentence aggregation mode and at trace level when streaming tokens (TextAggregationMode.TOKEN), where the accumulated turn text is already logged at debug level at flush time. The duplicate per-se...

v1.5.0

Added

• Added TogetherSTTService and TogetherTTSService for real-time speech-to-text and text-to-speech using Together AI's WebSocket APIs.
  (PR #4054)

• Added per-sentence synthesis mode and zero-shot audio prompt support to NvidiaTTSService, letting NVIDIA TTS users choose between stitched and per-request synthesis flows and configure voice-cloning prompts for supported models.
  (PR #4742)

• Added on_heartbeat_timeout event handler to PipelineWorker, fired when a heartbeat frame is not received within the monitor timeout period.
  (PR #4761)

• Added Time To First Audio (TTFA) metrics to TTS services, reported as TTFAMetricsData alongside the existing TTFB metric. TTFA measures the time to the first audible sample — TTFB plus the leading silence many providers pad onto the start of a response — so comparing the two shows how much perceived latency is padding versus service response time. Audible onset is detected from short-time RMS energy (detect_speech_onset in pipecat.audio.utils), which rejects noise-floor blips and brief transients; the MetricsLogObserver surfaces the new metric.
  (PR #4782)

• GeminiTTSService can now use the Gemini Developer API (google-genai) backend in addition to the existing Google Cloud backend. Pass api_key (or set GOOGLE_API_KEY) to authenticate with an API key instead of Google Cloud service-account credentials.

  • The backend is selected automatically: passing api_key opts into the GenAI backend, while credentials/credentials_path continue to use the Google Cloud backend. Use use_genai=True/False to force a backend explicitly. A GOOGLE_API_KEY present in the environment alone does not switch backends — it is only used once the GenAI backend is active.
  • New http_options parameter forwards google.genai.types.HttpOptions to the GenAI client.
  • The GenAI backend does not support prompt/style instructions or multi_speaker output; setting them logs a warning and they are ignored. Use the Google Cloud backend for those features.
    (PR #4787)

• Added a mode streaming parameter to AssemblyAISTTService, exposing AssemblyAI's U3 Pro latency/accuracy preset (min_latency, balanced, or max_accuracy). It trades transcription accuracy against turn-finalization latency and is only applicable to U3 Pro models, where the server defaults to balanced.
  (PR #4810)

• TaskManager can now be constructed with an event loop and an optional contextvars.Context (TaskManager(loop=..., context=...)), and creates all of its tasks within that context. You can pass a single task manager to WorkerRunner(task_manager=...) (and to individual workers) to share one loop and context across the runner and every worker, so context variables set in one task are visible to the others.
  (PR #4815)

• Added tunable parameters to the xAI TTS services: speed, optimize_streaming_latency, and text_normalization (plus with_timestamps on the WebSocket service). Set them via the service's Settings, e.g. XAITTSService.Settings(speed=1.1).
  (PR #4821)

• Added word-level timestamps to XAITTSService. When with_timestamps is enabled (now the default), xAI's per-character timing is converted into per-word TTSTextFrame objects, each carrying an accurate pts. Note that xAI delivers timestamps in coarse batches, so word frames are emitted in bursts; consumers should schedule off pts rather than arrival time.
  (PR #4821)

• Added a base_url parameter to TwilioFrameSerializer to configure the REST API host used for auto hang-up. By default the host is still derived from region/edge (unchanged behavior), but setting base_url lets you target a Twilio-API-compatible backend or a self-hosted server instead of api.twilio.com.
  (PR #4845)

• Added a first-class RTVI dtmf client message. Sending {type: "dtmf", data: {button: "1"}} makes the RTVIProcessor push an InputDTMFFrame downstream, the same path a telephony transport's keypress takes, so any bot with DTMF handling (e.g. a DTMFAggregator) reacts to it. One keypress per message.
  (PR #4849)

• Added DTMF keypress support to the behavioral evals. A scenario turn can now press keys with a dtmf: field (e.g. dtmf: "123#") instead of user:, sent as one RTVI dtmf message per key. A bot running a DTMFAggregator reacts to them as a transcription, so a dtmf turn can assert on user_transcription and response like a spoken turn.
  (PR #4849)

• Added built-in text transform functions for TTS voice formatting under
  pipecat.utils.text.transforms: strip_markdown, normalize_acronyms,
  expand_currency, expand_numbers, expand_percentages,
  expand_phone_numbers, expand_units, email_to_speech, normalize_dates,
  and replace_text. These can be composed individually via the
  text_transforms parameter on any TTSService, or used together via the new
  VoiceFormatter bundle.

  • VoiceFormatter is a single configurable callable that applies all
    transforms in the correct order (structural cleanup → language expansions →
    custom replacements). Most transforms are enabled by default; pass keyword
    arguments to toggle them:

    tts = CartesiaTTSService(
        text_transforms=[("*", VoiceFormatter(expand_numbers=True,

  normalize_acronyms=False))],
  )
  ```

  • Individual transforms can be composed for fine-grained control:

    tts = CartesiaTTSService(
        text_transforms=[("*", strip_markdown), ("*", expand_currency), ("*",

  expand_percentages)],
  )
  ```
  (PR #4854)

• Added silence-based keepalive to NvidiaSTTService to keep idle NVIDIA streaming ASR sessions from going stale. When no audio arrives for a while, the service sends silence over the existing stream instead of letting it sit idle and degrade.
  (PR #4877)

• Pipecat Flows is now part of pipecat-ai. The conversation-flow framework previously published as the separate pipecat-ai-flows package now ships with Pipecat under the pipecat.flows namespace — from pipecat.flows import FlowManager, NodeConfig — so there is no longer a separate package to install or keep version-matched. Code importing from pipecat_flows should switch to pipecat.flows. If the deprecated pipecat-ai-flows package is still installed alongside this Pipecat, Pipecat logs an error prompting you to remove it. The standalone package's release history remains available in the archived pipecat-flows repository.
  (PR #4882)

• Added clear_after_secs parameter to SOXRStreamAudioResampler (default 0.2) to control how long after inactivity the internal resampler state is cleared. Set to None to disable clearing.
  (PR #4886)

• Added resampler_clear_after_secs to FrameSerializer.InputParams so all telephony serializers (Twilio, Plivo, Vonage, Telnyx, Exotel, Genesys) expose this setting to callers.
  (PR #4886)

• Added a language_code streaming parameter to AssemblyAISTTService for declaring the audio language (e.g. "es", "fr"). On U3 Pro models a tier-1 code (en/es/fr/de/it/pt) steers transcription toward that language. It is mutually exclusive with language_detection and is not sent unless set, so existing behavior is unchanged.
  (PR #4889)

• Added AudioBufferStartRecordingFrame and AudioBufferStopRecordingFrame control frames. Push them through the pipeline to start and stop AudioBufferProcessor recording. The start_recording() / stop_recording() methods continue to work.
  (PR #4890)

• Added an auto_start_recording option to AudioBufferProcessor that starts recording as soon as the pipeline starts. Bots generated by the Pipecat CLI with the recording feature now use this option.
  (PR #4890)

• Added on_recording_started and on_recording_stopped events to AudioBufferProcessor, fired when recording starts and stops. on_recording_stopped fires after the final buffered audio has been emitted.
  (PR #4890)

• AI services can now describe themselves to downstream processors at start by overriding service_metadata_frame() to return a populated ServiceMetadataFrame; broadcast_service_metadata() broadcasts whatever it returns. The STT services that do server-side end-of-turn detection (Deepgram Flux, Cartesia Turns, AssemblyAI, Gladia, Speechmatics) use this to recommend ExternalUserTurnStrategies, so bots no longer need to set user_turn_strategies by hand; your own setting still wins.
  (PR #4892)

• Added endpoint_latency_adjustment_level to SonioxSTTService.Settings, exposing Soniox's endpoint-detection latency control (integer 0–3; higher finalizes turns sooner at some cost to acc...

v1.4.0

Added

• Added on_user_turn_message_added event handler on LLMUserAggregator, with a new UserTurnMessageAddedMessage arg type. It fires when the user aggregator writes a message to the LLM context, carrying the finalized turn text. In cascade mode it coincides with on_user_turn_stopped; in realtime mode (when realtime_service_mode=True on the aggregator pair) it's the canonical way to subscribe to "context just updated, here's the user text" (since the on_user_turn_stopped event fires before the message is finalized, with UserTurnStoppedMessage.content=None). Note that there's been no change to on_assistant_turn_stopped.
  (PR #4533)

• Added RealtimeServiceMetadataFrame, broadcast at pipeline start by realtime LLM services (OpenAI Realtime, Azure Realtime, Inworld, Grok/xAI Realtime, Gemini Live, AWS Nova Sonic, Ultravox). This frame can be used by other processors in the pipeline to configure themselves accordingly. Today, it only advertises two things: that a realtime service is present in the pipeline (indicated by the fact that the frame is sent at all), and emits_user_turn_frames, which says whether the realtime service can emit its own UserStartedSpeakingFrame and UserStoppedSpeakingFrames (suggesting local VAD/turn detection may not be needed in the pipeline).
  (PR #4533)

• Added to our examples "locally-driven-turns" variants for:

  • OpenAI Realtime (realtime-openai-locally-driven-turns.py)
  • Grok Realtime (realtime-grok-locally-driven-turns.py)
  • Inworld Realtime (realtime-inworld-locally-driven-turns.py)

  These join realtime-gemini-live-locally-driven-turns.py in showing how to configure each realtime service so that its turn-taking is dictated by local turn detection (e.g. VAD + smart turn analyzer).
  (PR #4533)

• Added a startup WARNING log on realtime LLM services that don't emit UserStartedSpeakingFrame/UserStoppedSpeakingFrame (Gemini Live, AWS Nova Sonic, Ultravox). The log is meant to draw attention to a couple of things:

  • That other processors in the pipeline (e.g. RTVI) may expect turn frames, and that the developer can enable local VAD/turn detection to supply them, and, relatedly
  • That when using local turn detection, local turns may NOT perfectly align with the "ground truth" of server-decided turns, so they should be thought of as APPROXIMATE (unless local turn detection is driving the realtime service's turns, in which case there's no separate server-decided ground truth)

  (The warning also serves as a little nudge to the realtime service providers: providing a "ground truth" signal of when the provider thinks the user has started or stopped speaking is very helpful to app developers!)
  (PR #4533)

• Added a realtime_service_mode: bool kwarg on LLMContextAggregatorPair, for opting into a set of behaviors tailored for use with realtime (speech-to-speech) services. Setting realtime_service_mode=True does three things: 1. Decouples context writes from the UserStoppedSpeakingFrame signal. Instead, the assistant response start triggers the user message writes. This ensures that context is written properly even when the realtime service provides no turn frames and local turn detection (i.e. local VAD) is disabled. This mechanism also enables the next point. 2. Lets UserStoppedSpeakingFrame fire without waiting for transcripts. When local turn detection is configured to drive realtime service conversations, UserStoppedSpeakingFrame is the signal that triggers assistant responses. By letting this frame fire earlier, we reduce latency. 3. Replaces the default turn strategies with ExternalUserTurnStartStrategy and ExternalUserTurnStopStrategy when the realtime service advertises that it emits its own turn frames. Various realtime services (OpenAI Realtime, Azure, Grok, Inworld) emit their own turn frames; in that case the External strategies fire on_user_turn_started / on_user_turn_stopped from the server-emitted UserStartedSpeakingFrame / UserStoppedSpeakingFrame. For realtime services that don't emit those frames — either because they never do (Gemini Live, Nova Sonic, Ultravox) or because server-side turn detection has been disabled at runtime (e.g. OpenAI Realtime with turn_detection=False, in locally-driven-turns setups) — the defaults stay in place so locally-driven turn detection (e.g. local VAD) can fire the events. Passing custom user_turn_strategies opts out of the swap.

  Note that when realtime_service_mode=True, you should listen for the new on_user_turn_message_added event to get the newly-added user message rather than on_user_turn_stopped, which no longer carries it.
  (PR #4533)

• Added private_endpoint parameter to AzureTTSService and AzureHttpTTSService for connecting via Private Link or custom domain endpoints, matching existing AzureSTTService support.
  (PR #4549)

• Added will_be_spoken field to AggregatedTextFrame. Set to True by the TTS service just before synthesis, allowing downstream processors and observers to know whether TTS will speak a given text segment before audio begins.
  (PR #4559)

• Added AggregatedTextProgressFrame — a new frame emitted alongside each TTSTextFrame during word-timestamp playback. It carries accumulated_text (text already spoken) and remaining_text (text not yet spoken) for the active segment, enabling downstream consumers such as the RTVI observer to do word-level highlighting without coupling to internal sequencer state.
  (PR #4559)

• Added AICQuailVADAnalyzer (pipecat.audio.vad.aic_quail_vad), a noise-robustVoice Activity Detection analyzer powered by the standalone Quail VAD 2.0 model from the ai-coustics SDK (aic-sdk~=2.3.0). It owns its own Processor and works independently of AICFilter, so it can sit before or after enhancement in the pipeline. Defaults to the published quail-vad-2.0-xxs-16khz model; supply model_id/model_path to override.
  (PR #4588)

• Added continuous_partials and interruption_delay connection parameters to the AssemblyAI streaming STT service (u3-rt-pro only). continuous_partials defaults to True so voice agents receive interim transcripts at a steady cadence during long turns; interruption_delay (0–1000 ms) overrides how soon the first partial is emitted. Both are exposed via AssemblyAISTTService.Settings and are omitted for non-u3-rt-pro models.
  (PR #4593)

• Added a user_audio_preroll_secs parameter to GeminiLiveLLMService controlling how much "pre-roll" audio is replayed (sent to Gemini Live) when the user turn start is confirmed, in locally-driven-turns mode (server-side VAD disabled). Defaults to None, auto-sizing the pre-roll duration from the upstream VAD's start_secs (which assumes VAD drives turn starts); set it explicitly when using a non-VAD turn-start strategy.
  (PR #4597)

• Added a user_audio_preroll_secs parameter to OpenAIRealtimeLLMService controlling how much "pre-roll" audio is replayed (re-appended to the input audio buffer) when the user turn start is confirmed, in locally-driven-turns mode (server-side turn detection disabled). Defaults to None, auto-sizing the pre-roll duration from the upstream VAD's start_secs (which assumes VAD drives turn starts); set it explicitly when using a non-VAD turn-start strategy.
  (PR #4599)

• Added word-level timestamp support to SmallestTTSService. Enabled by default via the word_timestamps constructor argument, it emits per-word TTSTextFrames aligned to audio playback so downstream consumers (captions, lip-sync, RTVI) receive word timing. Timestamps from each TTS request are offset onto the turn's continuous playback timeline, so multi-sentence turns stay correctly ordered. Available on Smallest's word-timestamp-capable voices; other voices simply emit no word events, so leaving it on is safe. Pass word_timestamps=False to fall back to whole-text frames.
  (PR #4612)

• Added a profanity setting to AzureSTTService (via settings=AzureSTTService.Settings(profanity=...)) controlling how Azure handles profanity in transcripts. Accepts "raw" (no masking), "masked" (Azure default, replaces profane words with ****), or "removed" (drops profane words). Defaults to None (keeps the Azure SDK default of "masked"). Use "raw" for non-English deployments where Azure's profanity list over-eagerly masks ordinary words. The setting is runtime-updatable and triggers a reconnect when changed.
  (PR #4620)

• WhatsApp connection_callback now receives the full call metadata (WhatsAppConnectCall) as a second argument, available in bot code via runner_args.body. This gives bots access to the caller's phone number, call ID, direction, and timestamp without any extra API calls.
  (PR #4622)

  Added the pipecat create project-scaffolding CLI to pipecat-ai, available via the optional cli extra. Install it with uv tool install "pipecat-ai[cli]" (add --with pipecatcloud to enable pipecat cloud), then run pipecat create to scaffold a new bot project. The CLI dependencies are optional, so they are not pulled into a plain `pip install pipe...

v1.3.0

• Pipecat pipelines are multi-agent compatible by default. The new multi-agent framework (pipecat.workers) turns every PipelineWorker (previously PipelineTask) into a peer on a shared bus that passes typed messages, dispatches @job work, and coordinates with siblings, while existing single-pipeline code keeps running untouched. examples/multi-worker/ ships ready-to-run patterns: LLM handoff, parallel debate, sidecar code assistants and hardware controllers, distributed deployments over Redis or PGMQ, point-to-point WebSocket proxies, and UI workers driving a web client over RTVI.
  (PR #4493)

• Added UIWorker (pipecat.workers.ui): an LLM worker that observes and drives a client web UI over the RTVI UI channel — for voice agents that act on what the user is looking at. It reads the page's accessibility snapshots, routes client UI events to @ui_event handlers, drives the page with UI commands (scroll_to, highlight, select_text, click, set_input_value), and answers screen-grounded questions. PipelineWorker connects it to the client automatically when RTVI is enabled — no extra wiring.

  • A voice agent delegates a turn via the built-in respond job; the worker returns an answer for the voice LLM to speak, or speaks it verbatim through the agent's TTS with respond_to_job(answer, tts_speak=True).
  • ReplyToolMixin provides a ready-made reply tool (a spoken answer plus the standard UI actions).
  • ui_job_group(...) fans work out to peer workers, surfaced to the client as cancellable progress cards.
  • UI_STATE_PROMPT_GUIDE is drop-in system-prompt text that teaches the LLM the <ui_state> wire format.
    (PR #4540)

• Added VonageVideoConnectorTransport, a new transport integration for real-time Vonage WebRTC sessions using the Vonage Video Connector library.
  (PR #4052)

• Added InceptionLLMService for Inception's Mercury 2 diffusion reasoning model, with support for reasoning_effort and realtime settings.
  (PR #4423)

• Added plain WebSocket transport support to the development runner. Bots can now accept connections from non-telephony WebSocket clients (e.g., browser apps using protobuf framing) via the /ws-client endpoint alongside other transports.
  (PR #4442)

• Added GET /status endpoint to the development runner that reports which transports the running instance accepts (all by default, or the single transport passed via -t).
  (PR #4442)

• Added support for the Rime coda TTS model to RimeTTSService and RimeHttpTTSService. The temperature, top_p, and repetition_penalty settings are not used by coda. Also added a timeScaleFactor setting (for the arcana and coda models) to both services — values above 1.0 slow down audio playback; values below 1.0 speed it up.
  (PR #4511)

• Added max_endpoint_delay_ms to SonioxSTTService.Settings, controlling the maximum delay (500-3000 ms) before endpoint detection finalizes a turn.
  (PR #4521)

• Added LLMService.append_system_instruction(...): append durable text to a service's system instruction so it's included on every inference and survives context resets.
  (PR #4540)

• Added CartesiaTurnsSTTService for streaming speech-to-text against the Cartesia Streaming ASR v2 (Ink-2) turn-based WebSocket endpoint (/stt/turns/websocket). The server drives turn boundaries via turn.start / turn.update / turn.end messages, which the service translates into UserStartedSpeakingFrame, finalized TranscriptionFrame, and UserStoppedSpeakingFrame. Eager end-of-turn predictions and turn resumes (turn.eager_end and turn.resume) are surfaced via the on_turn_eager_end and on_turn_resume event handlers.
  (PR #4552)

• Added the STTService.supports_ttfs property, which subclasses can override to return False when TTFS doesn't apply to their architecture (e.g. turn-based STTs where the server defines turn boundaries). When False, STTMetadataFrame is broadcast with ttfs_p99_latency=0.0 and the "ttfs_p99_latency not set" warning is suppressed.
  (PR #4585)

Changed

• ⚠️ The development runner now supports all transports (WebRTC, Daily, telephony, plain WebSocket) simultaneously from a single server. The /start endpoint accepts a "transport" field to select the transport per-request; omitting -t at startup enables all transports instead of defaulting to WebRTC. The Daily browser-redirect route moved from GET / to GET /daily.
  (PR #4442)

• Changed the default model for RimeTTSService and RimeHttpTTSService from arcana to coda. Code that relied on the implicit default should set model="arcana" explicitly to preserve previous behavior.
  (PR #4511)

• OpenRouter LLM service now defaults to openai/gpt-4.1.
  (PR #4513)

• OpenRouter LLM requests now convert developer messages to user messages by default for broader model compatibility. Override this by subclassing OpenRouterLLMService or setting llm.supports_developer_role = True for models that support the developer role.
  (PR #4513)

• SonioxSTTService now applies settings updates (e.g. via STTUpdateSettingsFrame) using a graceful reconnect instead of a hard disconnect/reconnect, preserving the service's reconnect retry behavior.
  (PR #4521)

• Updated the default p99 TTFS latency values for Smallest AI, Mistral, and XAI STT so turn stop timing uses measured values instead of the conservative fallback.
  (PR #4522)

• Updated the development runner startup banner to show the prebuilt client URL once and list enabled or disabled transports with install hints.
  (PR #4524)

• Services and transports with missing optional dependencies now raise ImportError instead of a bare Exception when their module is imported without the required extra installed. The original ModuleNotFoundError is preserved as __cause__, so code that wraps these imports can now use except ImportError: cleanly instead of except Exception:.
  (PR #4525)

• Bumped pipecat-ai-prebuilt to 1.0.1 in the runner extra, updating the prebuilt client UI served by the development runner.
  (PR #4531)

• Replaced the transformers.WhisperFeatureExtractor dependency in LocalSmartTurnAnalyzerV3 with a vendored numpy-only implementation, reducing peak RSS at import from ~566 MB to ~60 MB and cold-start time from ~5.0 s to ~0.3 s. Behavior is numerically equivalent (matches the reference numpy code path within 1e-5 absolute tolerance; ONNX model output is bit-identical on representative inputs).

  • Smart Turn v3 no longer imports transformers at module load.
  • Prepares the ground for making transformers an optional dependency in a future release.
  • The vendored STFT is vectorized via numpy.lib.stride_tricks.sliding_window_view + batched np.fft.rfft, cutting _power_spectrogram runtime by ~55% (~4.0 ms → ~1.8 ms per call on a typical 8-second segment at 16 kHz) while preserving the same parity tolerances against the reference implementation.
    (PR #4536)

• ⚠️ Renamed the RTVI UI Worker Protocol's vocabulary from the pipecat-subagents task/agent terms to Pipecat's native job/worker. This spans the wire messages (ui-task → ui-job-group, ui-cancel-task → ui-cancel-job-group), their envelope kinds and fields (task_id → job_id, agents/agent_name → workers/worker_name), the paired Python models/frames (UITask* → UIJobGroup*, RTVIUITask*Frame → RTVIUIJobGroup*Frame), and the @pipecat-ai/client-js / client-react APIs (RTVIEvent.UITask → UIJobGroup, cancelUITask → cancelUIJobGroup, useUITasks → useUIJobGroups, UITasksProvider → UIJobGroupsProvider). These primitives shipped in 1.2.0 but were never documented, so no real consumers are affected.
  (PR #4540)

• transformers is no longer a base dependency, so pip install pipecat-ai no longer pulls it in. This follows Smart Turn v3 dropping its transformers import; the only remaining users (the deprecated LocalSmartTurnAnalyzerV2/CoreML analyzers and the Moondream service) already require the local-smart-turn and moondream extras, which continue to install transformers.
  (PR #4546)

• Widened the deepgram extra to deepgram-sdk>=6.1.1,<8 so installations can resolve to either deepgram-sdk 6.x or 7.x. DeepgramSTTService now handles the agent_rest keyword argument that deepgram-sdk 7.2.0 added to DeepgramClientEnvironment, so custom base_url configuration keeps working on both 6.x and 7.x.
  (PR #4565)

• Dropped the upper bound on the websockets-base extra (websockets>=13.1) so downstream deployments can resolve to websockets 16.x and beyond. Pipecat's websockets usage relies ...

v1.2.1

Changed

• Changed the default WebSocket endpoints for GradiumSTTService and GradiumTTSService to the region-neutral wss://api.gradium.ai/api/speech/asr and wss://api.gradium.ai/api/speech/tts. Gradium now automatically routes traffic to the nearest endpoint. Override the url to pin to a specific region.
  (PR #4500)

Fixed

• Fixed bot hangs when filter_incomplete_user_turns was enabled and the LLM responded by calling a tool. The user turn never finalized, so the assistant aggregator gated the tool-result context push and the LLM continuation never ran. Tool calls now finalize the turn the moment they start, before the function dispatches.
  (PR #4501)

v1.2.0

Added

• Added a session_id field to RunnerArguments so bots can log or trace a per-session identifier in local development the same way they can in Pipecat Cloud. The development runner now mints a UUID at every construction site, and paths that already returned a sessionId to the caller (Daily /start, dial-in webhook) share that same UUID with the runner args instead of generating two. The SmallWebRTC /api/offer endpoint also accepts an optional session_id query parameter so the /sessions/{session_id}/... proxy can thread it through.
  (PR #4385)

• Added a max_buffer_delay_ms constructor argument to CartesiaTTSService for controlling Cartesia's server-side text buffering. When unset, Pipecat picks a sensible default based on text_aggregation_mode: 0 in SENTENCE mode (custom buffering — avoids stacking client-side aggregation on top of Cartesia's default 3000ms server buffer) and unset in TOKEN mode (Cartesia's managed buffering applies). Pass an explicit value (0–5000ms) to override.
  (PR #4390)

• Added a mip_opt_out constructor argument to DeepgramTTSService and DeepgramHttpTTSService so callers can opt out of the Deepgram Model Improvement Program. When set, the value is forwarded to Deepgram as a query parameter on the speak request. Defaults to None, which preserves the existing behavior. See https://dpgr.am/deepgram-mip for pricing implications before enabling.
  (PR #4400)

• Added an opt-in add_tool_change_messages flag to the LLM aggregators (set via LLMContextAggregatorPair(..., add_tool_change_messages=True)) that appends a developer-role message to the context whenever LLMSetToolsFrame changes the set of advertised standard tools. Helps the LLM stay coherent across mid-conversation tool changes, mitigating several flavors of tool-call-related hallucination: calling tools that have been removed, avoiding tools that have been re-added, and hallucinating output (made-up answers or tool-call-shaped non-tool-calls) when tools are unavailable.
  (PR #4404)

• Added deferred(strategy) and DeferredUserTurnStopStrategy in pipecat.turns.user_stop. Wraps a stop strategy so it fires only the inference-triggered event and suppresses on_user_turn_stopped, leaving finalization to another strategy in the chain such as LLMTurnCompletionUserTurnStopStrategy.
  (PR #4405)

• Added ExternalUserTurnCompletionStopStrategy in pipecat.turns.user_stop — a generic stop strategy that finalizes the user turn whenever a UserTurnInferenceCompletedFrame arrives, regardless of which component produced it. LLMTurnCompletionUserTurnStopStrategy now extends this base; future producers (Flux, custom end-of-turn classifiers, etc.) can use the base directly or subclass it to add producer-specific setup.
  (PR #4405)

• Added on_user_turn_inference_triggered, a new event on the user turn controller, processor, aggregator and stop strategies that fires when a strategy has enough signal to start LLM inference. By default it fires together with on_user_turn_stopped; a gating strategy can fire only the inference-triggered event and defer finalization to a peer.
  (PR #4405)

• Added FilterIncompleteUserTurnStrategies in pipecat.turns.user_turn_strategies — a UserTurnStrategies specialization that wraps the detector chain with deferred(...) and appends LLMTurnCompletionUserTurnStopStrategy as the finalizer. Common case: user_turn_strategies=FilterIncompleteUserTurnStrategies(). Pass config=UserTurnCompletionConfig(...) to customize timeouts and prompts.
  (PR #4405)

• Added LLMTurnCompletionUserTurnStopStrategy in pipecat.turns.user_stop. When installed, the strategy gates on_user_turn_stopped on a UserTurnInferenceCompletedFrame (a new fieldless system frame emitted by any component that can judge turn completeness — e.g. the UserTurnCompletionLLMServiceMixin on ✓). A finalization_timeout provides a safety net if no completion frame ever arrives.
  (PR #4405)

• Added first-class RTVI support for the UI Agent Protocol:

  • Adds ui-event, ui-snapshot, and ui-cancel-task client-to-server messages, plus ui-command and ui-task server-to-client messages, with paired *Data / *Message pydantic models.
  • Adds built-in command payload models for Toast, Navigate, ScrollTo, Highlight, Focus, Click, SetInputValue, and SelectText; matching default handlers live in @pipecat-ai/client-react.
  • Adds RTVIProcessor.on_ui_message for inbound ui-event, ui-snapshot, and ui-cancel-task messages.
  • Adds five UI pipeline frames, mirroring the client-message frame-and-event pattern: downstream code pushes RTVIUICommandFrame / RTVIUITaskFrame for the observer to wrap into outbound UICommandMessage / UITaskMessage envelopes, while the processor pushes inbound RTVIUIEventFrame, RTVIUISnapshotFrame, and RTVIUICancelTaskFrame alongside on_ui_message.
  • Bumps the RTVI PROTOCOL_VERSION from 1.2.0 to 1.3.0.
    (PR #4407)

• AWS Transcribe STT, Polly TTS, Bedrock LLM, and the Bedrock AgentCore processor now resolve credentials via the standard boto3 provider chain (EC2 instance profiles, EKS pod roles / IRSA, ECS task roles, SSO, ~/.aws/credentials) when explicit credentials and AWS_* environment variables are absent. Services running with IAM roles no longer need to export static credentials.
  (PR #4416)

• Added keyterms support to ElevenLabs STT services so Scribe V2 callers can bias transcription for both file-based and realtime transcription.
  (PR #4426)

• Added watchdog_min_timeout parameter to DeepgramFluxSTT and DeepgramFluxSageMakerSTT (default 0.5 seconds) to control the minimum silence duration before the watchdog sends a silence packet to prevent dangling turns. The actual threshold is max(chunk_duration * 2, watchdog_min_timeout), so it also adapts automatically to the audio chunk size in use.
  (PR #4430)

• Added cancel_on_interruption=False support for GeminiLiveLLMService on models that support Gemini's NON_BLOCKING tool mechanism (currently Gemini 2.x); the conversation now continues while the tool runs. On models that don't yet support NON_BLOCKING (Gemini 3.x), the service surfaces a one-time warning explaining the limitation. (Note: an intermittent 1008 error can occasionally fire on Gemini 2.5 during long-running tool calls; we auto-reconnect.)
  (PR #4448)

• Added NvidiaSageMakerWebsocketSTTService for streaming speech recognition using NVIDIA Nemotron ASR via an AWS SageMaker bidirectional-stream endpoint. Produces InterimTranscriptionFrame and TranscriptionFrame frames, is VAD-aware, and automatically reconnects on error.
  (PR #4464)

• Added NVIDIA Magpie TTS services via AWS SageMaker: NvidiaSageMakerHTTPTTSService (single HTTP invocation, streams raw PCM back) and NvidiaSageMakerWebsocketTTSService (persistent HTTP/2 bidi-stream with full interruption support via InterruptibleTTSService).
  (PR #4464)

• Added support for reasoning configuration on OpenAIRealtimeLLMService, for use with reasoning-capable Realtime models such as gpt-realtime-2.
  (PR #4470)

• Inworld TTS updates:

  • Added delivery_mode setting (STABLE/BALANCED/CREATIVE) to InworldTTSService and InworldHttpTTSService, enabling the stability-vs-creativity tradeoff in inworld-tts-2.
  • Added language support to InworldTTSService and InworldHttpTTSService. The language setting is now forwarded to the API, and a new language_to_inworld_language() helper normalizes Pipecat Language enums to Inworld's BCP-47 locale tags.
    (PR #4473)

Changed

• Updated the default SonioxTTSService model from tts-rt-v1-preview to the generally available tts-rt-v1.
  (PR #4386)

• Default cartesia_version for CartesiaTTSService bumped from 2025-04-16 to 2026-03-01, matching CartesiaHttpTTSService and unlocking the use_normalized_timestamps and max_buffer_delay_ms fields.
  (PR #4390)

• ⚠️ CartesiaTTSService now sends use_normalized_timestamps: true instead of the deprecated use_original_timestamps field. Word timestamps now reflect what was actually spoken (post text-normalization and pronunciation-dictionary substitution), matching the convention Pipecat uses for ElevenLabs. This is a behavior change for sonic-3 users, who were previously receiving timestamps tied to the input transcript.
  (PR #4390)

• Broadened tool_resources to app_resources for easy access not just in tool handlers but in other places like custom FrameProcessors. Three changes: a rename (tool_resources → app_resources), a new app_resources property on PipelineTask, and a new pipeline_task property on FrameProcessor. Tool handlers now read params.app_resources; custom processors read `self.pipeline_task.ap...

v1.1.0

Added

• Added MistralSTTService for real-time speech-to-text using Mistral's Voxtral Realtime API (voxtral-mini-transcribe-realtime-2602). Supports streaming transcription with interim results, automatic language detection, and VAD-driven utterance lifecycle.
  (PR #4253)

• Added buttons field to OutputDTMFFrame and OutputDTMFUrgentFrame for sending multi-key DTMF sequences as a list[KeypadEntry]. Use OutputDTMFFrame.from_string("123#") (or the equivalent on OutputDTMFUrgentFrame) to build one from a dial string, and to_string() to convert back.
  (PR #4313)

• Added DailyTransport.send_dtmf() to expose the Daily call client's DTMF sending capability, enabling applications to send tones during a call (e.g. IVR navigation).
  (PR #4313)

• Added DailyOutputDTMFFrame and DailyOutputDTMFUrgentFrame frames. In addition to the inherited buttons, they accept session_id, digit_duration_ms and method, which are forwarded to Daily's send_dtmf as sessionId, digitDurationMs and method.
  (PR #4313)

• Added incremental pyright type checking. A pyrightconfig.json at the repo root uses typeCheckingMode: "basic" with an explicit include list of modules that pass cleanly (clocks, metrics, transcriptions, frames, observers, extensions, turns, pipeline, runner). Remaining modules will be added in subsequent PRs. CI enforces the checked set via uv run pyright in the format workflow.
  (PR #4324)

• Added multilingual support to DeepgramFluxSTTService via a new language_hints: list[Language] setting. Works with Deepgram's new flux-general-multi model to bias transcription across English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch. Omit the hints to use auto-detection, or pass a subset to bias toward expected languages. Hints can be updated mid-stream via STTUpdateSettingsFrame (sent as a Deepgram Configure control message, no reconnect) to support detect-then-lock flows.
  (PR #4326)

• Added fine-grained server-side VAD tuning options to SarvamSTTService.Settings for the saaras:v3 model, including speech thresholds, frame-count controls, pre-speech padding, interruption sensitivity, and initial-frame skipping.
  (PR #4334)

• Added XAISTTService for real-time speech-to-text using xAI's voice STT WebSocket API (wss://api.x.ai/v1/stt). Streams raw audio (PCM, µ-law, or A-law) and emits interim and final transcription frames driven by the server's is_final / speech_final flags. Settings expose interim_results, endpointing, language, multichannel, channels, and diarize. Requires the xai optional extra (pip install "pipecat-ai[xai]").
  (PR #4340)

• Added XAITTSService for streaming text-to-speech using xAI's WebSocket TTS endpoint (wss://api.x.ai/v1/tts). Streams text.delta chunks up and base64 audio.delta chunks down on the same connection so audio begins flowing before the full utterance finishes synthesizing; complements the batch-HTTP XAIHttpTTSService. Defaults to raw PCM output so TTSAudioRawFrame needs no decoding. The xai optional extra now pulls in pipecat-ai[websockets-base].
  (PR #4341)

• Added SonioxTTSService, a real-time WebSocket TTS service that streams text in and audio out over a persistent connection. Install with pip install "pipecat-ai[soniox]".
  (PR #4360)

• Added support for Daily's built-in screenVideo destination in DailyTransport. When "screenVideo" is included in video_out_destinations transport parameter, a dedicated screen video track is created at join time and frames with transport_destination="screenVideo" are routed to it.

  params = DailyParams(
        video_out_enabled=True,
        video_out_is_live=True,
        video_out_width=1280,
        video_out_height=720,
        video_out_destinations=["screenVideo"]
  )
  
  ...
  
  frame = OutputImageRawFrame(...)
  frame.transport_destination = "screenVideo"

  (PR #4370)

• Added camera_out_send_settings to DailyParams. This dict is passed verbatim to the Daily client's camera publishing settings, allowing applications to fully control encoding, codec, bitrate, and framerate.

  params = DailyParams(
      camera_out_send_settings={
          "maxQuality": "high",
          "encodings": {
              "high": {"maxBitrate": 2_000_000, "maxFramerate": 30}
          },
      },
  )

  (PR #4370)

• Added tool_resources to PipelineTask and FunctionCallParams. Pass an application-defined object (DB handles, clients, state, etc.) to PipelineTask(..., tool_resources=...) and access it from any tool handler via params.tool_resources. Passed by reference; the caller retains their handle and can read mutations after the task finishes. Resolves #4256.
  (PR #4371)

Changed

• Updated NVIDIA STT services to align with Nemotron Speech defaults and
  configuration: api_key is now optional for local deployments, additional
  recognition settings are available (including alternatives, word offsets, and
  diarization), and streaming/segmented docs now reflect Nemotron Speech APIs.

  • NVIDIA streaming STT now sets TranscriptionFrame.finalized=True when the provider marks a result as final, and preserves language on both TranscriptionFrame and InterimTranscriptionFrame.
    (PR #4269)

• Updated NvidiaLLMService to emit model reasoning as LLMThought*Frames (from both reasoning_content and <think>...</think> output), avoid mixing reasoning text into normal assistant content, and allow keyless local NIM endpoints while warning when the cloud endpoint is used without an API key.
  (PR #4270)

• STT services now reconnect safely when settings change: reconnection is deferred until the current user turn ends (i.e., until UserStoppedSpeakingFrame is received) rather than interrupting an active speech session. Audio frames received while the reconnect is in progress are buffered and replayed once the new connection is ready. CartesiaSTTService and DeepgramSTTService both use this new behavior.
  (PR #4311)

• Reduced debug log noise for LLM services. The system instruction is now logged once when composed (e.g. when turn completion is enabled) instead of on every LLM call. Per-call logs now show only the conversation messages, consistent across Google, Anthropic, AWS, and OpenAI services.
  (PR #4314)

• LiveKitRunnerArguments.token is now a required str (previously str | None with a default of None). LiveKit requires a token to join a room, so the type now reflects reality. This only affects custom runners that construct LiveKitRunnerArguments directly; code consuming the argument from the standard runner is unaffected.
  (PR #4324)

• TranscriptionFrame.language and InterimTranscriptionFrame.language emitted by DeepgramFluxSTTService now reflect the language Deepgram detected for each turn (read from the languages field on Flux's TurnInfo event). On flux-general-multi this gives per-turn accuracy for downstream consumers (e.g. TTS voice selection). flux-general-en continues to emit Language.EN.
  (PR #4326)

• Added includes_inter_frame_spaces parameter to
  TTSService.add_word_timestamps and _add_word_timestamps (default None).
  When True, downstream consumers will not inject additional spaces between
  tokens; None leaves each frame's own default unchanged.

  • InworldTTSService now passes includes_inter_frame_spaces=True when reporting word timestamps, since Inworld tokens already include inter-word spacing.
    (PR #4330)

• SarvamSTTService now uses saaras:v3 as its default model instead of saarika:v2.5. Applications that relied on the previous default should set settings=SarvamSTTService.Settings(model="saarika:v2.5") explicitly.
  (PR #4334)

• SpeechTimeoutUserTurnStopStrategy now waits only user_speech_timeout when a transcript arrives without a VAD stop event, rather than max(ttfs_p99_latency, user_speech_timeout). If you had ttfs_p99_latency > user_speech_timeout, turn detection in that path is slightly faster than before.
  (PR #4337)

• If you use an STT service that emits finalized transcripts (Speechmatics, Soniox, Deepgram Flux, AssemblyAI) with SpeechTimeoutUserTurnStopStrategy, user turns now end as soon as user_speech_timeout elapses after VAD stop. Previously the strategy also waited for the STT P99 latency (ttfs_p99_latency) even when the transcript was already marked final. user_speech_timeout is still honored as a floor — STT finalization never shortens it.
  (PR #4337)

• ⚠️ `PlivoFrameSerializer...

v1.0.0

Migration guide: https://docs.pipecat.ai/pipecat/migration/migration-1.0

Added

• Updated LemonSlice transport:

  • Added on_avatar_connected and on_avatar_disconnected events triggered when the avatar joins and leaves the room.
  • Added api_url parameter to LemonSliceNewSessionRequest to allow overriding the LemonSlice API endpoint.
  • Added support for passing arbitrary named parameters to the LemonSlice API endpoint.
    (PR #3995)

• Added Inworld Realtime LLM service with WebSocket-based cascade STT/LLM/TTS, semantic VAD, function calling, and Router support.
  (PR #4140)

• ⚠️ Added WebSocket-based OpenAIResponsesLLMService as the new default for the OpenAI Responses API. It maintains a persistent connection to wss://api.openai.com/v1/responses and automatically uses previous_response_id to send only incremental context, falling back to full context on reconnection or cache miss. The previous HTTP-based implementation is now available as OpenAIResponsesHttpLLMService.
  (PR #4141)

• Added group_parallel_tools parameter to LLMService (default True). When True, all function calls from the same LLM response batch share a group ID and the LLM is triggered exactly once after the last call completes. Set to False to trigger inference independently for each function call result as it arrives.
  (PR #4217)

• Added async function call support to register_function() and register_direct_function() via cancel_on_interruption=False. When set to False, the LLM continues the conversation immediately without waiting for the function result. The result is injected back into the context as a developer message once available, triggering a new LLM inference at that point.
  (PR #4217)

• Added enable_prompt_caching setting to AWSBedrockLLMService for Bedrock ConverseStream prompt caching.
  (PR #4219)

• Added support for streaming intermediate results from async function calls. Call result_callback multiple times with properties=FunctionCallResultProperties(is_final=False) to push incremental updates, then call it once more (with is_final=True, the default) to deliver the final result. Only valid for functions registered with cancel_on_interruption=False.
  (PR #4230)

• Added LLMMessagesTransformFrame to facilitate programmatically editing context in a frame-based way.

  The previous approach required the caller to directly grab a reference to the context object, grab a "snapshot" of its messages at that point in time, transform the messages, and then push an LLMMessagesUpdateFrame with the transformed messages. This approach can lead to problems: what if there had already been a change to the context queued in the pipeline? The transformed messages would simply overwrite it without consideration.
  (PR #4231)

• The development runner now exports a module-level app FastAPI instance (from pipecat.runner.run import app) so you can register custom routes before calling main().
  (PR #4234)

• ToolsSchema now accepts custom_tools for OpenAI LLM services (OpenAILLMService, OpenAIResponsesLLMService, OpenAIResponsesHttpLLMService, and OpenAIRealtimeLLMService), letting you pass provider-specific tools like tool_search alongside standard function tools.
  (PR #4248)

• Added enhancements to NvidiaTTSService:

  • Cross-sentence stitching: multiple sentences within an LLM turn are fed into a single SynthesizeOnline gRPC stream for seamless audio across sentence boundaries (requires Magpie TTS model v1.7.0+).
  • custom_dictionary and encoding parameters for IPA-based custom pronunciation and output audio encoding.
  • Metrics generation (can_generate_metrics returns true) and stop_all_metrics() when an audio context is interrupted.
  • gRPC error handling around synthesis config retrieval (GetRivaSynthesisConfig).
    (PR #4249)

• Added MistralTTSService for streaming text-to-speech using Mistral's Voxtral TTS API (voxtral-mini-tts-2603). Supports SSE-based audio streaming with automatic resampling from the API's native 24kHz to any requested sample rate. Requires the mistral optional extra (pip install pipecat-ai[mistral]).
  (PR #4251)

• Added truncate_large_values parameter to LLMContext.get_messages(). When True, returns compact deep copies of messages with binary data (base64 images, audio) replaced by short placeholders and long string values in LLM-specific messages recursively truncated. Useful for serialization, logging, and debugging tools.
  (PR #4272)

• CartesiaSTTService now supports runtime settings updates (e.g. changing language or model via STTUpdateSettingsFrame). The service automatically reconnects with the new parameters. Previously, settings updates were silently ignored.
  (PR #4282)

• Added pcm_32000 and pcm_48000 sample rate support to ElevenLabs TTS services.
  (PR #4293)

• Added enable_logging parameter to ElevenLabsHttpTTSService. Set to False to enable zero retention mode (enterprise only).
  (PR #4293)

Changed

• Updated onnxruntime from 1.23.2 to 1.24.3, adding support for Python 3.14.
  (PR #3984)

• MCPClient now requires async with MCPClient(...) as mcp: or explicit start()/close() calls to manage the connection lifecycle.
  (PR #4034)

• ⚠️ Updated langchain extra to require langchain 1.x (from 0.3.x), langchain-community 0.4.x (from 0.3.x), and langchain-openai 1.x (from 0.3.x). If you pin these packages in your project, update your pins accordingly.
  (PR #4192)

• WebsocketService reconnection errors are now non-fatal. When a websocket service exhausts its reconnection attempts (either via exponential backoff or quick failure detection), it emits a non-fatal ErrorFrame instead of a fatal one. This allows application-level failover (e.g. ServiceSwitcher) to handle the failure instead of killing the entire pipeline.
  (PR #4201)

• Changed GrokLLMService default model from grok-3-beta to grok-3, now that the model is generally available.
  (PR #4209)

• GoogleImageGenService now defaults to imagen-4.0-generate-001 (previously imagen-3.0-generate-002).
  (PR #4213)

• ⚠️ BaseOpenAILLMService.get_chat_completions() now accepts an LLMContext instead of OpenAILLMInvocationParams. If you override this method, update your signature accordingly.
  (PR #4215)

• When multiple function calls are returned in a single LLM response, by default (when group_parallel_tools=True) the LLM is now triggered exactly once after the last call in the batch completes, rather than waiting for all function calls.
  (PR #4217)

• ⚠️ LLMService.function_call_timeout_secs now defaults to None instead of 10.0. Deferred function calls will run indefinitely unless a timeout is explicitly set at the service level or per-call. If you relied on the previous 10-second default, pass function_call_timeout_secs=10.0 explicitly.
  (PR #4224)

• Updated NvidiaTTSService:

  • Made api_key optional for local NIM deployments.
  • Voice, language, and quality can be updated without reconnecting the gRPC client; new values take effect on the next synthesis turn, not for the current turn's in-flight requests.
  • Replaced per-sentence synchronous synthesize_online calls with async queue-backed gRPC streaming.
  • Streaming now uses asyncio tasks with explicit gRPC cancellation on interruption and stale-response filtering when a stream is aborted or replaced.
  • Renamed Riva references to Nemotron Speech in docs and messages.
  • Disabled automatic TTS start frames at the service level (push_start_frame=False) and emit TTSStartedFrame when a stitched synthesis stream is started for a context.
    (PR #4249)

Removed

• ⚠️ Removed OpenPipeLLMService and the openpipe extra. OpenPipe was acquired by CoreWeave and the package is no longer maintained. If you were using openpipe as an LLM provider, switch to the underlying provider directly (e.g. openai). The OpenPipe interface can still be used with OpenAILLMService by specifying a base_url.
  (PR #4191)

• ⚠️ Removed NoisereduceFilter. Use system-level noise reduction or a service-based alternative instead.
  (PR #4204)

• ⚠️ Removed deprecated vad_enabled and vad_audio_passthrough transport params.
  (PR #4204)

• ⚠️ Removed deprecated camera_in_enabled, camera_in_is_live, camera_in_width, camera_in_height, `came...

v0.0.108

Added

• Added SarvamLLMService with support for sarvam-30b, sarvam-30b-16k, sarvam-105b and sarvam-105b-32k.
  (PR #3978)

• Added on_turn_context_created(context_id) hook to TTSService. Override this to perform provider-specific setup (e.g. eagerly opening a server-side context) before text starts flowing. Called each time a new turn context ID is created.
  (PR #4013)

• Added XAIHttpTTSService for text-to-speech using xAI's HTTP TTS API.
  (PR #4031)

• Added support for "developer" role messages in conversation context across all LLM adapters. For non-OpenAI services (Anthropic, Google, AWS Bedrock), "developer" messages are converted to "user" messages (use system_instruction to set the system instruction). For OpenAI services, "developer" messages pass through in conversation history. For the Responses API, they are kept as "developer" role (matching the existing "system" → "developer" conversion).
  (PR #4089)

• Added SmallestTTSService, a WebSocket-based TTS service integration with Smallest AI's Waves API. Supports the Lightning v2 and v3.1 models with configurable voice, language, speed, consistency, similarity, and enhancement settings.
  (PR #4092)

• Added warnings in turn stop strategies when VADParams.stop_secs differs from the recommended default (0.2s) or when stop_secs >= STT p99 latency, which collapses the STT wait timeout to 0s and may cause delayed turn detection. The warnings guide developers to re-run the stt-benchmark with their VAD settings.
  (PR #4115)

• Added domain parameter to AssemblyAISTTSettings for specialized recognition modes such as Medical Mode (domain="medical-v1").
  (PR #4117)

• Added NovitaLLMService for using Novita AI's LLM models via their OpenAI-compatible API.
  (PR #4119)

• Added cleanup() method to VADAnalyzer and VADController so VAD analyzer resources are properly released when no longer needed. Custom VADAnalyzer subclasses can override cleanup() to free any held resources.
  (PR #4120)

• Added on_end_of_turn event handler to AssemblyAISTTService. This fires after the final transcript is pushed, providing a reliable hook for end-of-turn logic that doesn't race with TranscriptionFrame. Works in both Pipecat and AssemblyAI turn detection modes.
  (PR #4128)

• Added DeepgramFluxSageMakerSTTService for running Deepgram Flux speech-to-text on AWS SageMaker endpoints. Use with ExternalUserTurnStrategies to take advantage of Flux's turn detection.
  (PR #4143)

• Added Mem0MemoryService.get_memories() convenience method for retrieving all stored memories outside the pipeline (e.g. to build a personalized greeting at connection time). This avoids the need to manually handle client type branching, filter construction, and async wrapping.
  (PR #4156)

Changed

• Added context prewarming path for InworldTTSService to improve first audio latency.
  (PR #4013)

• Added KrispVivaVadAnalyzer for Voice Activity Detection using the Krisp VIVA SDK (requires krisp_audio).
  (PR #4022)

• Modified InworldTTSService to close context at end of turn instead of relying on idle timeout. (PR #4028)

• Added Gemini 3 support to the Gemini Live service.
  (PR #4078)

• TTSService: the default stop_frame_timeout_s (idle time before an automatic TTSStoppedFrame is pushed when push_stop_frames=True) has changed from 2.0 to 3.0 seconds.
  (PR #4084)

• ⚠️ GeminiLLMAdapter now only treats messages[0] as the initial system message, matching all other adapters. Previously it searched for the first "system" message anywhere in the conversation history. A "system" message appearing later in the list will now be converted to "user" instead of being extracted as the system instruction.

  (PR #4089)

• Fixed InworldTtsService to fallback to full text when TTS timestamps are not received.
  (PR #4113)

• ⚠️ Realtime services (Gemini Live, OpenAI Realtime, Grok Realtime, Nova Sonic) now prefer system_instruction from service settings over an initial system message in the LLM context, matching the behavior of non-realtime services. Previously, context-provided system instructions took precedence. A warning is now logged when both are set.
  (PR #4130)

• Bumped nvidia-riva-client minimum version to >=2.25.1.
  (PR #4136)

• Upgraded protobuf from 5.x to 6.x (>=6.31.1,<7).
  (PR #4136)

• Unrecognized language strings (e.g. Deepgram's "multi") no longer produce a warning at startup. The log message has been downgraded to debug level since these are valid service-specific values that are passed through correctly.
  (PR #4137)

• GrokLLMService and GrokRealtimeLLMService now live in the pipecat.services.xai module alongside XAIHttpTTSService, since all three use the same xAI API. Update imports from pipecat.services.grok.* to pipecat.services.xai.* (e.g. from pipecat.services.xai.llm import GrokLLMService).
  (PR #4142)

• ⚠️ Bumped mem0ai dependency from ~=0.1.94 to >=1.0.8,<2. Users of the mem0 extra will need to update their mem0ai package.
  (PR #4156)

Deprecated

• pipecat.services.grok.llm, pipecat.services.grok.realtime.llm, and
  pipecat.services.grok.realtime.events are deprecated. The old import paths
  still work but emit a DeprecationWarning; use pipecat.services.xai.llm,
  pipecat.services.xai.realtime.llm, and
  pipecat.services.xai.realtime.events instead.
  (PR #4142)

Removed

• ⚠️ TTSService.add_word_timestamps() no longer supports the "Reset" and "TTSStoppedFrame" sentinel strings. If you have a custom TTS service that called await self.add_word_timestamps([("Reset", 0)]) or await self.add_word_timestamps([("TTSStoppedFrame", 0), ("Reset", 0)], ctx_id), replace them with await self.append_to_audio_context(ctx_id, TTSStoppedFrame(context_id=ctx_id)) and let _handle_audio_context manage the word-timestamp reset automatically.
  (PR #4145)

• Removed SambaNovaSTTService. SambaNova no longer offers speech-to-text audio models. Use another STT provider instead.
  (PR #4154)

Fixed

• Fixed Gemini Live (GoogleGeminiLiveLLMService) not honoring settings.system_instruction. The system instruction was being read from a deprecated constructor parameter instead of the settings object, causing it to be silently ignored.
  (PR #4089)

• Fixed AWSBedrockLLMAdapter sending an empty message list to the API when the only message in context was a system message. The lone system message is now converted to "user" role instead of being extracted, matching the existing Anthropic adapter behavior.
  (PR #4089)

• Fixed Gemini Live pipeline hanging indefinitely when an EndFrame was deferred while waiting for the bot to finish responding and turn_complete never arrived. As a possible root-cause fix, turn_complete messages are now handled even if they lack usage_metadata. As a fallback, the deferred EndFrame now has a 30-second safety timeout.
  (PR #4125)

• Fixed ElevenLabs WebSocket disconnections (1008 "Maximum simultaneous contexts exceeded") caused by rapid user interruptions. When interruptions arrived before any TTS text was generated, phantom contexts were created on the ElevenLabs server that were never closed, eventually exceeding the 5-context limit.
  (PR #4126)

• Fixed the final sentence being dropped from the conversation context when using RTVI text input with non-word-timestamp TTS services. The LLMFullResponseEndFrame was racing ahead of the last TTSTextFrame, causing the LLMAssistantAggregator to finalize the context before the final sentence arrived.
  (PR #4127)

• Fixed audio crackling and popping in recordings when both user and bot are speaking. AudioBufferProcessor no longer injects silence into a track's buffer while that track is actively producing audio, preventing mid-utterance interruptions in the recorded output.
  (PR #4135)

• Fixed websocket TTS word timestamps so interrupted contexts cannot leak stale words or backward PTS values into later turns.
  (PR #4145)
  ...