feat(telnyx): add Telnyx TTS plugin - #621
Conversation
Streams text to speech over the Telnyx WebSocket endpoint and decodes the returned MP3 into PcmData as it arrives. Telnyx closes the socket after each stop frame, so the plugin connects per synthesis rather than holding one socket open. Follows the plugins/sarvam precedent of one vendor plugin covering LLM, STT, and TTS.
📝 WalkthroughWalkthroughTightens the Telnyx TTS provider’s WebSocket lifecycle with handshake timeouts, synchronized cancellation state, and improved close-error logging. Incoming messages now require dictionary payloads, invalid base64 is skipped, and undecodable MP3 payloads are dropped after logging. TTS and its exception are exported publicly. Documentation and runtime dependencies are updated, with unit coverage for ID3 handling, configuration, malformed payloads, decoding failures, streaming output, and reconnection. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
plugins/telnyx/vision_agents/plugins/telnyx/tts.py (1)
133-139: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
close()doesn't tear down the WebSocket directly.
close()closesself._sessionbut never callsself._close_ws()or resetsself._ws. Ifclose()is called while a synthesis'sfinallyblock hasn't run yet (e.g. the consuming async generator was abandoned rather than exhausted), the staleself._wsreference outlives the session closing it implicitly rather than via its own graceful close path.♻️ Proposed fix
async def close(self) -> None: """Close the current WebSocket and release the aiohttp session.""" await super().close() + await self._close_ws() if self._session is not None and not self._session.closed: await self._session.close() self._session = None self._on_disconnected()
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d419cc24-2770-4040-a99a-2db31de5795e
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
plugins/telnyx/README.mdplugins/telnyx/pyproject.tomlplugins/telnyx/tests/test_telnyx_tts.pyplugins/telnyx/vision_agents/plugins/telnyx/__init__.pyplugins/telnyx/vision_agents/plugins/telnyx/tts.py
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/telnyx/vision_agents/plugins/telnyx/tts.py (1)
142-143: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the unneeded
Anyannotations from the catch-all overrides.
tts.TTS.stream_audiouses*_, **__, not*_: Any, **__: Any; keep the Telnyx override matching that signature and drop the guideline-violating annotations.Source: Path instructions
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ca362830-c00b-4149-a246-5843f3ffb18d
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
plugins/telnyx/README.mdplugins/telnyx/pyproject.tomlplugins/telnyx/tests/test_telnyx_tts.pyplugins/telnyx/vision_agents/plugins/telnyx/__init__.pyplugins/telnyx/vision_agents/plugins/telnyx/tts.py
🚧 Files skipped from review as they are similar to previous changes (2)
- plugins/telnyx/pyproject.toml
- plugins/telnyx/README.md
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
plugins/telnyx/tests/test_telnyx_tts.py (1)
94-108: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse a typed pytest fixture for WebSocket setup.
fake_ws()is test setup encoded as a static helper, and the new async functions omit return annotations. Move setup into a typed@pytest.fixturefactory and annotatereceive,parse, and test coroutines.As per coding guidelines: “Use pytest.fixture for test setup, not helper methods” and “Use type annotations everywhere.”
Also applies to: 110-131
Source: Coding guidelines
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 67bd4608-6251-4d5f-b2fa-f269118ef306
📒 Files selected for processing (2)
plugins/telnyx/tests/test_telnyx_tts.pyplugins/telnyx/vision_agents/plugins/telnyx/tts.py
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/telnyx/tests/test_telnyx_tts.py (1)
94-116: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftExercise barge-in through the real cancellation path.
DroppedWS/FakeSessionmock transport and set_stop_eventdirectly, so this never invokesstop_audio()or validates its close behavior. Use a typed pytest fixture with a local aiohttp WebSocket, callawait tts.stop_audio(), and assert the stream ends cleanly.Also applies to: 166-170
Source: Coding guidelines
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 484c472c-01fe-47dd-bc61-7cfd2cad9f8f
📒 Files selected for processing (2)
plugins/telnyx/tests/test_telnyx_tts.pyplugins/telnyx/vision_agents/plugins/telnyx/tts.py
| ws = await self._connect() | ||
| # Telnyx rejects a text frame that is not preceded by an | ||
| # init frame with "Invalid message". | ||
| await ws.send_str(json.dumps({"text": " "})) | ||
| await ws.send_str(json.dumps({"text": text})) | ||
| await ws.send_str(json.dumps({"text": ""})) | ||
| async for chunk in self._receive_audio(ws): | ||
| yield chunk |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Cancel pending connection attempts on barge-in.
stop_audio() can run while _connect() is awaiting the handshake. _ws is still None, so _close_ws() cannot interrupt it; once connected, this code still sends initialization and text frames. A hung handshake delays cancellation until idle_timeout. Race the connect task against _stop_event.wait(), cancel and await the connect task when stop wins, and skip sends.
🧰 Tools
🪛 ast-grep (0.45.0)
[info] 160-160: use jsonify instead of json.dumps for JSON output
Context: json.dumps({"text": " "})
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 161-161: use jsonify instead of json.dumps for JSON output
Context: json.dumps({"text": text})
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 162-162: use jsonify instead of json.dumps for JSON output
Context: json.dumps({"text": ""})
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
1 participant
Adds streaming Telnyx TTS to
plugins/telnyx/, which today only contains themedia streaming transport from #594. The pattern precedent is
plugins/sarvam/,which keeps
llm.py,stt.py, andtts.pyin one vendor plugin. LLM and STTare separate PRs so each lands independently.
What it adds
plugins/telnyx/vision_agents/plugins/telnyx/tts.pyTTSandTelnyxTTSErrorexports in the plugin__init__.pyaiohttpdependency in the pluginpyproject.tomlplugins/telnyx/tests/test_telnyx_tts.pyTwo wire protocol details worth reviewing
Both were found by probing the live endpoint, and both differ from what the
sarvam TTS shape would suggest.
One connection per synthesis. Sarvam holds a single socket open across
stream_audiocalls. Telnyx closes the socket once it has served the emptytext stop frame, and a second synthesis on the same socket gets a close frame
back, so the plugin reconnects each time. A synthesis is also rejected with
Invalid messageunless a{"text": " "}init frame precedes the real text.MP3 files are concatenated, not a single bitstream. A synthesis can span
several MP3 files, each introduced by its own ID3v2 tag at the head of a
WebSocket frame. Feeding the concatenated bytes straight to a decoder fails
part way through the utterance, which truncates the audio rather than raising
anything obvious at the call site.
_Id3Stripperdrops those tags, includingtags whose header or body spans WebSocket frames.
The decoded sample rate depends on the voice: Polly voices return 24 kHz and
Kokoro voices 22.05 kHz. The plugin takes the rate from the decoder instead of
exposing a
sample_rateargument that could disagree with the audio.Decoding uses
av, which is already anagents-coredependency, so this addsno new decode dependency.
Testing
Verified against the live Telnyx API, not from docs alone.
uv run ruff check .andruff format --check .passuv run dev.py mypyanddev.py mypy-pluginspassuv run dev.py validate-extraspassesuv run pytest -m "not integration" plugins/telnyx/tests/passes, 42 tests,including 8 covering the ID3 stripper across frame boundaries
uv run pytest -m integration plugins/telnyx/tests/test_telnyx_tts.pypasseswith a real
TELNYX_API_KEY, 3 tests. One asserts the decoded audio runs pastthree seconds and is non silent, which is what catches a regression in the
ID3 handling, and one asserts a second synthesis reconnects successfully.
uv.lockchanges by two lines, for the new dependency.Open question
idle_timeoutdefaults to 10 seconds as a safety net for the case where theserver never sends
isFinal. Happy to change the default if there is a housevalue for this.
Relation to the other two Telnyx PRs
This is one of three independent PRs, each branched off
main, not stacked:#620 LLM, #621 TTS, #622 STT. Each touches the plugin
__init__.py,pyproject.toml,README.md, and two lines ofuv.lock, so whichever landsfirst leaves the other two with small conflicts in those four files. Happy to
rebase on request, or to fold all three into one PR if you would rather review
them together.