Skip to content

2.2.25 Backend Speech To Speech

Jump to bottom
av edited this page Aug 2, 2026 · 1 revision

Speech To Speech

Handle: s2s
URL: http://localhost:35020

Speech To Speech is an open, modular speech-to-speech pipeline from Hugging Face: Voice Activity Detection, Speech To Text, an LLM turn, and Text To Speech, streamed end-to-end in realtime. Harbor runs it as a backend service exposing an OpenAI Realtime-compatible WebSocket endpoint (/v1/realtime) and routes the LLM step to any OpenAI-compatible backend.

  • Configurable STT (parakeet-tdt, whisper, faster-whisper) and TTS (qwen3, kokoro, pocket, chattts, facebook-mms) components
  • LLM step served by an OpenAI-compatible backend of your choice (llama.cpp by default)
  • GPU (CUDA) and CPU support

Starting

There is no official pre-built image - Harbor builds the service from the upstream repository:

# Build the image (first time, and after changing s2s.git.ref)
harbor build s2s

# Start the service
harbor up s2s

The first start downloads the STT/TTS model weights into the shared HuggingFace cache, which may take a while - the container healthcheck allows up to two minutes of startup time. Track progress via:

harbor logs s2s

If Nvidia Docker Toolkit support is detected on the host, Harbor applies the Nvidia overlay (compose.x.s2s.nvidia.yml) automatically to run the pipeline on the GPU.

Usage

The service speaks the OpenAI Realtime protocol over WebSocket at ws://localhost:35020/v1/realtime. The upstream repository ships a reference client:

# From a checkout of https://github.com/huggingface/speech-to-speech
python scripts/listen_and_play_realtime.py --host localhost --port 35020

The client captures your microphone, streams audio to the pipeline, and plays back the spoken reply.

llama.cpp integration

When started together with llamacpp, Harbor applies compose.x.s2s.llamacpp.yml and points the LLM step at the llama.cpp server automatically:

harbor up s2s llamacpp

To use a different OpenAI-compatible backend, set s2s.openai.url and s2s.openai.key accordingly.

Configuration

Environment Variables

Following options can be set via harbor config:

# Host port for the realtime WebSocket endpoint
harbor config get s2s.host.port         # 35020

# Git ref the image is built from
harbor config get s2s.git.ref           # https://github.com/huggingface/speech-to-speech.git#main

# OpenAI-compatible backend for the LLM step
harbor config get s2s.openai.url        # http://llamacpp:8080/v1
harbor config get s2s.openai.key        # sk-s2s

# Model name to request from the backend
# Empty by default - llama.cpp's router discovers models automatically
harbor config get s2s.model

# STT component: parakeet-tdt (default), whisper, faster-whisper
harbor config get s2s.stt               # parakeet-tdt

# TTS component: qwen3 (default), kokoro, pocket, chattts, facebook-mms
harbor config get s2s.tts               # qwen3

# System prompt for the LLM turn
harbor config get s2s.system.prompt     # "You are a helpful assistant."

# Extra CLI arguments appended to the speech-to-speech command
harbor config get s2s.extra.args

# Additional volumes to mount into the container
harbor config get s2s.volumes

See all s2s config options:

harbor config ls | grep -i s2s

Service-specific environment overrides can also be set via harbor env:

harbor env s2s <key> <value>

Volumes

  • The global HuggingFace cache (HARBOR_HF_CACHE) is mounted to /root/.cache/huggingface so STT/TTS weights are shared with other Harbor services and persist across restarts. When the global cache is not configured, ./services/s2s/.hf-cache is used instead.
  • Extra mounts can be added via s2s.volumes.

Troubleshooting

  • The service is WebSocket-only - opening http://localhost:35020 in a browser will not show a UI. Use the realtime client above or any OpenAI Realtime-compatible client.
  • If the container is repeatedly restarted as unhealthy on first boot, the model downloads likely exceeded the healthcheck grace period - check harbor logs s2s and simply let it finish; subsequent starts reuse the cache.
  • If the LLM step fails, verify the backend is reachable from the container: harbor exec s2s python -c "import urllib.request; print(urllib.request.urlopen('http://llamacpp:8080/v1/models').status)" (adjust for your s2s.openai.url).
  • To rebuild against a different upstream revision: harbor config set s2s.git.ref "https://github.com/huggingface/speech-to-speech.git#<ref>", then harbor build s2s.

Links

Home | CLI Reference | Services | Adding New Service | Compatibility

Clone this wiki locally