A telephone-based conversational agent built with Pipecat, powered by OpenAI's APIs and Twilio. Ask it about NCAA basketball scores during March Madness!
Rename the env.example file to .env and set the following:
OPENAI_API_KEYto use OpenAI (obviously)DAILY_API_KEYOptional, but highly recommended. Enables easy Daily integration (see below). Get it from your Pipecat Cloud Dashboard:https://pipecat.daily.co/<your-org-id>/settings/daily
You'll need a Docker Hub account to deploy. You'll also need a Twilio account if you want to call your bot.
There are two botfiles in this repo. bot-llm.py uses the traditional STT to LLM to STT cascading flow. bot-s2s.py Uses the speech-to-speech model from the OpenAI Realtime API.
The S2S model usually responds more quickly than the cascading flow. But the cascading approach gives you more control over how the bot operates. For example, OpenAI's new GPT-4o-powered transcription service allows you to prompt the transcriber to improve accuracy. In bot-llm.py, we're instructing the transcriber to listen for basketball-themed words. You also get the full power of the GPT-4o LLM and more configurable text-to-speech. It's worth trying both bots to see the differences.
Taken from the Pipecat Cloud Quickstart.
Set up a local environment:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtChoose which version of the bot you want to deploy:
# use s2s:
cp bot-s2s.py bot.py
# or use the cascading bot:
cp bot-llm.py bot.pyBuild the docker image:
docker build --platform=linux/arm64 -t openai-twilio:latest .
docker tag openai-twilio:latest your-username/openai-twilio:0.1
docker push your-username/openai-twilio:0.1Deploy it to Pipecat Cloud:
pcc auth login # to authenticate
pcc secrets set openai-twilio-secrets --file .env # to store your environment variables
pcc deploy openai-twilio your-username/openai-twilio:0.1 --secrets openai-twilio-secrets
To connect this agent to Twilio:
-
Purchase a number from Twilio, if you haven't already
-
Collect your Pipecat Cloud organization name:
pcc organizations listYou'll use this information in the next step.
- Create a TwiML Bin:
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Connect>
<Stream url="wss://api.pipecat.daily.co/ws/twilio">
<Parameter name="_pipecatCloudServiceHost" value="AGENT_NAME.ORGANIZATION_NAME"/>
</Stream>
</Connect>
</Response>where:
- AGENT_NAME is your agent's name (the name you used when deploying)
- ORGANIZATION_NAME is the value returned in the previous step
In this case, it will look something like value="openai-twilio.level-gorilla-gray-123".
- Assign the TwiML Bin to your phone number:
- Select your number from the Twilio dashboard
- In the
Configuretab, setA call comes intoTwiML Bin - Set
TwiML Binto the Bin you created in the previous step - Save your configuration
Now call your Twilio number, and you should be connected to your bot!
Pipecat uses Transports to handle audio and video to and from the bot. When you use the FastAPIWebsocketTransport and deploy your bot to Pipecat Cloud, the service handles connecting Twilio's websocket to your bot. This is tricky to do locally, so this bot also has a DailyTransport.
If you've set DAILY_API_KEY in your .env file, you can run the bot locally in your Python environment with:
LOCAL_RUN=1 python bot.py # (or bot-llm.py or bot-s2s.py)If you look at the console output, you should see something like:
2025-03-18 22:23:13.309 | INFO | runner:configure_with_args:79 - Daily room URL: https://cloud-longstring.daily.co/another-long-string
Open that URL in your browser, and you'll join a Daily room with your bot.
Additionally, with the DailyTransport in your botfile, you can talk to the bot in the Pipecat Cloud dashboard: https://pipecat.daily.co/<your-org-id>/agents/openai-twilio/sandbox.
Modify the system prompt in bot.py:
instructions="""You are a helpful and friendly AI...Search for basketball_scores in the codebase to find where the existing function calls are registered. Learn all about Pipecat function calling here.