A FreeSWITCH module that streams live call audio to your AI agent over a WebSocket — and plays the agent's voice back — full duplex, with ten ready-made protocol adapters, module-side turn detection, agent-driven call control, PCI masking, multi-stream fan-out, and built-in latency metrics.
Features · Why · Quickstart · API · Architecture · Roadmap · Security · Contributing
Every FreeSWITCH voice-AI write-up hits the same wall: the incumbent module (mod_audio_stream) has
playback (agent → caller) that doesn't work reliably, forcing uuid_broadcast hacks. Earshot is a
drop-in mod_audio_stream alternative whose WRITE_REPLACE path is validated with real audio, full
duplex, on one leg. That's the wedge — everything else builds on it. The full pitch is in
WHY.md; the complete capability list is in FEATURES.md.
- 7 wire protocols, agent unmodified —
native,twilio,openai(Realtime),deepgram(Voice Agent),vapi,elevenlabs,gemini(Live),pipecat,assemblyai+cartesia(streaming STT). Swap vendors with one word. - Turn-taking for any agent — module-side VAD →
speech_started/speech_stopped, a ready-gate (no "answered into silence"), and speech-triggered barge-in. - Agent drives the call — opt-in, per-action control channel (
commands=play,hangup): transfer, hangup, DTMF, play, record, hold, bridge, setvar — with every argument validated (app-exec, path traversal, and exec-triggering variables rejected), audited viaearshot::command. - PCI/PII masking — mute audio + redact DTMF to the agent during card entry.
- Multi-stream fan-out — agent + live transcription + supervisor on one call.
- Latency KPIs — time-to-first-audio, per-turn response time, WebSocket RTT.
- Codecs + resampling — G.711 µ-law/a-law + L16, transparent 8k/16k/24k.
- Built on libwebsockets — the WebSocket transport rides the mature, high-performance libwebsockets library (not a hand-rolled RFC-6455 client), on a shared service-loop pool that scales with sessions, not threads.
- Reliable + observable — reconnect w/ jitter, bounded queue,
earshot::metricsevents, correlation by SIP Call-ID + channel UUID. - Drop-in migration —
uuid_audio_stream/audio_streamcompat runs mod_audio_stream dialplans unchanged.
Point proto= at your stack — same module, one word different. All adapters are
mock-tested in CI; ✅ marks those also validated end-to-end on a real SIP call (against the live
service, or — for the self-hosted Pipecat wire format — its official protobuf schema).
| Agent / protocol | proto= |
Status |
|---|---|---|
| OpenAI Realtime | openai |
✅ live-validated |
| Deepgram Voice Agent | deepgram |
✅ live-validated |
| Native (any WebSocket) | native |
✅ live-validated |
| Vapi (WebSocket transport) | vapi |
✅ live-validated |
| AssemblyAI (streaming STT) | assemblyai |
✅ live-validated · transcription (dir=in) |
| Cartesia (streaming STT) | cartesia |
✅ live-validated · transcription (dir=in) |
| Pipecat | pipecat |
✅ validated (protobuf) |
| ElevenLabs Conversational AI | elevenlabs |
✅ live-validated |
| Google Gemini Live | gemini |
✅ live-validated |
| Twilio Media Streams | twilio |
adapter ✓ · framing echo-tested |
mod_audio_stream hands you an audio pipe. In production the first question is "is the agent fast
enough — on every call, right now?" Earshot answers it with no agent instrumentation: every stream
emits an earshot::metrics event (periodic, on-close, or on-demand JSON) carrying the numbers voice
teams actually optimize:
- Time-to-first-audio — call start → the agent's first word. The single "does it feel alive?" number.
- Per-turn response latency (avg + max) — caller stops → agent starts; catch a slow model or vendor the moment it drifts.
- WebSocket RTT — transport health, sampled via ping/pong.
- Throughput & backpressure — tx/rx frames + bytes, play-buffer depth, queue drops, reconnects.
At one call it's a debugger; at ten thousand it's your fleet latency scoreboard — ship the events to Prometheus/OTel and alert on p95 time-to-first-audio per vendor. Every metric is keyed to the two-key trace (SIP Call-ID + channel UUID), so a single call stitches together across FreeSWITCH, your agent, and your logs. Lifecycle, DTMF, barge-in/turn, and agent-command events flow over the same FreeSWITCH event socket — the whole call is observable, the agent untouched.
Everything Earshot observes is published as FreeSWITCH custom events, so an application follows a
call without touching the media path, the agent, or the WebSocket. Nothing extra to run: FreeSWITCH's
Event Socket Layer (ESL) is already a live subscriber to the event bus, so any ESL client — fs_cli,
Node (modesl), Python (greenswitch), Go — receives them the instant they fire. Every event is tagged
with the channel Unique-ID and the correlation id, so you can act on one specific call.
Earshot fires nine typed events (subclass earshot::…):
| Event | Fires when | Key headers |
|---|---|---|
earshot::connected |
WebSocket handshake to the agent completes | url |
earshot::ready |
playback gate opens — the caller won't answer into silence | corr |
earshot::speech_started / …stopped |
module-side VAD marks a caller turn boundary | corr |
earshot::dtmf |
caller pressed a digit (redacted inside a PCI mask window) | digit | masked |
earshot::command |
the agent ran a whitelisted call-control action (audit trail) | action, api, ok, result |
earshot::metrics |
periodic / on-close / on-demand | latency KPIs + counters |
earshot::disconnected / …error |
the agent socket closed / failed | reason |
Subscribe from fs_cli (any ESL client uses the same wire command):
event plain CUSTOM earshot::metrics
A live earshot::metrics frame — exactly the headers your app reads:
Event-Subclass: earshot::metrics
corr: 4a9b8c…@carrier # = SIP Call-ID — the key that joins your logs
stream-id: default
first-audio-ms: 62 # call answered → agent's first word
response-ms: 340 # last caller-stop → agent-start, per turn
response-ms-max: 610
ws-rtt-ms: 1 # transport health, sampled via ping/pong
What you build with it — no agent instrumentation, no polling:
- a live "caller is speaking" UI or barge-in indicator (
speech_started/speech_stopped), - a per-call latency scoreboard / p95 alert (
metrics→ Prometheus/OTel), - a compliance audit log of every agent-driven action (
command) and DTMF (dtmf), - call routing / hand-off logic gated on
ready,disconnected, orerror.
Because every event carries the two-key trace (SIP Call-ID ↔ channel UUID), one call stitches
together across your load balancer, FreeSWITCH, the agent, and your logs — you write no correlation
plumbing. That's the step up from a raw audio pipe: where mod_audio_stream relays the agent's
messages as one generic event, Earshot emits a typed call lifecycle an application can act on directly.
1. Build & install (needs FreeSWITCH dev headers + libwebsockets-dev):
cmake -S . -B build && cmake --build build
sudo cmake --install build # -> /usr/lib/freeswitch/mod/mod_earshot.so
fs_cli -x "load mod_earshot"2. Point a call at your agent — dialplan:
<extension name="ai-agent">
<condition field="destination_number" expression="^5000$">
<action application="answer"/>
<action application="earshot" data="start ws://127.0.0.1:9099/ws proto=native vad=on vad_barge=on"/>
<action application="playback" data="silence_stream://-1"/> <!-- keep the leg up -->
</condition>
</extension>3. Any WebSocket agent works. A minimal echo agent (hear yourself talk):
# pip install websockets
import asyncio, websockets
async def handle(ws):
async for msg in ws: # binary caller audio (µ-law 8k by default)
await ws.send(msg) # ...echo it straight back to the caller
async def main():
async with websockets.serve(handle, "0.0.0.0", 9099):
await asyncio.Future()
asyncio.run(main())Call 5000 and you'll hear yourself — proving both directions. Swap the echo for your LLM,
or point proto= at OpenAI/Deepgram/ElevenLabs/Gemini/Pipecat and run your existing agent unchanged.
More recipes (pipecat, OpenAI Realtime, transcription fan-out, PCI masking) in QUICKSTART.md.
Dialplan app (operates on the current channel):
<action application="earshot" data="start ws://agent/ws proto=openai commands=true metrics=10"/>API (fs_cli / ESL — <uuid> first, then the verb):
earshot <uuid> start ws://agent/ws proto=native [id=<name>] [dir=in|out|both] ...
earshot <uuid> flush # barge-in: clear queued agent audio
earshot <uuid> send '{"type":"..."}'
earshot <uuid> mask on|off # PCI: mute audio + redact DTMF to the agent
earshot <uuid> status | metrics # per-stream JSON (add id=<name> for a fan-out fork)
earshot <uuid> stop
Fan-out — many streams on one channel:
<action application="earshot" data="start ws://agent/ws"/>
<action application="earshot" data="start ws://stt/ws id=transcribe dir=in proto=deepgram"/>
<action application="earshot" data="start ws://mon/ws id=supervisor dir=in"/>Full option/verb/event reference: FEATURES.md · API detail: docs/API.md · internals: docs/ARCHITECTURE.md.
cmake -S . -B build && cmake --build build # module (needs FS headers) + unit tests
ctest --test-dir build # codec / queue / protobuf unit tests (no FS needed)
cpack --config build/CPackConfig.cmake -G DEB # -> mod-earshot_*.deb (when FS headers are present)
docker build -t earshot-build . # reproducible build image (see Dockerfile)The portable core (codec, protobuf) unit-tests without FreeSWITCH, so CI stays
green on any runner; the module itself compiles where FS dev headers are available. See
.github/workflows/ci.yml.
Earshot is MIT and free to run. If you're taking it — or FreeSWITCH voice-AI generally — to production, the author (xpertvoip) offers paid help:
- Integration & POC — a working phone-call → AI-agent bridge on your stack, live in days, against the vendor of your choice (OpenAI, Deepgram, Pipecat, or your own WebSocket agent).
- Bridging your SIP infrastructure to AI voice — connect an existing FreeSWITCH / SIP carrier setup to real-time voice agents cleanly, with correlation, barge-in, and PCI-safe DTMF.
- Scaling to millions of calls a day — take a FreeSWITCH + AI-voice deployment horizontal: the shared-event-loop transport, fleet architecture, capacity planning, and latency/quality tuning so voice agents scale out under real load.
- Production support & retainers — on-call, upgrades, and SLA-backed help for teams running earshot or a FreeSWITCH voice-AI platform.
- Training & workshops — "Production Voice-AI on FreeSWITCH" for your team.
📧 xpertvoipai@gmail.com — tell me your stack and what you're building.
Earshot / mod_earshot is a working name (evokes "within earshot"). It's a single token across the
source and CMake — see the note at the top of src/mod_earshot.c to rename.
MIT — see LICENSE. Attribution and provenance in NOTICE. Contributions welcome — CONTRIBUTING.md.