title	Create a Voice Virtual Assistant
author	Alexandre Sajus
uid	TODO
datePublished	TODO
description	Learn how to build a Voice Virtual Assistant using ElevenLabs API and Python for seamless AI-powered interactions.
published	TODO
header	TODO
bannerImage	TODO
tags	intermediate python AI

Create a Voice Virtual Assistant

Prerequisites: Python fundamentals, API usage Versions: Python 3.11, python-dotenv 1.0.1, elevenlabs 1.54.0, elevenlabs[pyaudio] Read Time: 60 minutes

Introduction

Voice assistants like Siri, Google Assistant, and Alexa have revolutionized the way we interact with technology. In this tutorial, we’ll learn how to create a Voice Virtual Assistant using ElevenLabs API and Python. This assistant will be able to engage in natural conversations and provide helpful responses in real time.

The final assistant will:

• Process user voice and text input
• Use ElevenLabs API for voice synthesis
• Provide a seamless conversational experience

Here is a sneak peek of the final assistant in action:

Let's dive in!

Setting Up the Environment

1. Install Required Packages

Before we start, make sure you have Python installed. Then, install the required dependencies:

pip install elevenlabs elevenlabs[pyaudio] python-dotenv

Processing audio requires additional dependencies on Linux and MacOS:

• For Linux, you need to install portaudio19:

sudo apt install portaudio19

• For MacOS, you need to install portaudio:

brew install portaudio

2. Setting up ElevenLabs

ElevenLabs provides a Conversational AI API that we will use to create our Voice Assistant. - The API records the user's voice through the microphone

• 🎤 It processes it to know when the user has finished speaking or is interrupting the assistant
• 🤖 It calls an LLM model to generate a response
• 📈 It synthesizes the response into speech
• 🔊 It plays the synthesized speech through the speakers

1. Sign up at ElevenLabs and follow the instructions to create an account.

2. Once signed in, go to "Conversational AI"

3. Go to "Agents"

4. Click on "Start from blank"

5. Create a ".env" file at the root of your project folder. We will use this file to store our API credentials securely. This way they won't be hardcoded in the script. In this ".env" file, add your Agent ID:

AGENT_ID=your_agent_id

6. Go to the "Security" tab, enable the "First message" and "System prompt" overrides, and save. This will allow us to customize the assistant's first message and system prompt using Python code.

7. Click on your profile and go to "API keys". Create a new API key and copy it to your ".env" file:

API_KEY="sk_XXX...XXX"

Please make sure to save your ".env" file after adding the credentials.

ElevenLabs is now set up and ready to be used in our Python script!

Note: ElevenLabs works with a credit system. When you sign up, you get 10,000 free credits which amount to 15 minutes of conversation. You can buy more credits if needed.

Building the Voice Assistant

The full code for this application is available here.

1. Load Environment Variables

Create a Python file (e.g., voice_assistant.py) and load your API credentials:

import os
from dotenv import load_dotenv

load_dotenv()

AGENT_ID = os.getenv("AGENT_ID")
API_KEY = os.getenv("API_KEY")

2. Configure ElevenLabs Conversation API

We will set up the ElevenLabs client and configure a conversation instance.

We'll start by importing the necessary modules:

from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface
from elevenlabs.types import ConversationConfig

We will then configure the conversation with the agent's first message and system prompt. We are going to inform the assistant that the user has a schedule and prompt it to help the user. In this part you can customize:

• The user's name: what the assistant will call the user
• The schedule: the user's schedule that the assistant will use to provide help
• The prompt: the message that the assistant will receive when the conversation starts to understand the context of the conversation
• The first message: the first message the assistant will say to the user

Prompts are used to provide context to the assistant and help it understand the user's needs. They can be used to provide information about the user's environment, preferences, or any other information that can help the assistant provide better responses.

user_name = "Alex"
schedule = "Sales Meeting with Taipy at 10:00; Gym with Sophie at 17:00"
prompt = f"You are a helpful assistant. Your interlocutor has the following schedule: {schedule}."
first_message = f"Hello {user_name}, how can I help you today?"

We are then going to set this configuration to our ElevenLabs agent:

conversation_override = {
    "agent": {
        "prompt": {
            "prompt": prompt,
        },
        "first_message": first_message,
    },
}

config = ConversationConfig(
    conversation_config_override=conversation_override,
    extra_body={},
    dynamic_variables={},
)

client = ElevenLabs(api_key=API_KEY)
conversation = Conversation(
    client,
    AGENT_ID,
    config=config,
    requires_auth=True,
    audio_interface=DefaultAudioInterface(),
)

3. Implement Callbacks for Responses

To improve user experience, define callback functions to handle assistant responses. These functions will print the assistant's responses and user transcripts. We also define a function to handle the situation where the user interrupts the assistant:

def print_agent_response(response):
    print(f"Agent: {response}")

def print_interrupted_response(original, corrected):
    print(f"Agent interrupted, truncated response: {corrected}")

def print_user_transcript(transcript):
    print(f"User: {transcript}")

4. Start the Voice Assistant Session

Finally, initiate the conversation session:

conversation = Conversation(
    client,
    AGENT_ID,
    config=config,
    requires_auth=True,
    audio_interface=DefaultAudioInterface(),
    callback_agent_response=print_agent_response,
    callback_agent_response_correction=print_interrupted_response,
    callback_user_transcript=print_user_transcript,
)

conversation.start_session()

Running the Assistant

Please make sure your audio devices are correctly set up in your system settings before running the code.

Execute the script:

python voice_assistant.py

The assistant will start listening for input and responding in real time!

You can stop the assistant at any time by closing the terminal.

Conclusion

Congratulations! 🎉 You've successfully built a Voice Virtual Assistant using ElevenLabs API. You can extend its capabilities by integrating it with home automation, calendars, or other APIs to make it even more useful.

Stay creative and keep experimenting with AI-powered assistants!

More Resources

• ElevenLabs Conversational AI Overview
• ElevenLabs Python SDK Documentation
• Enable your assistant to execute Python functions with Client Tools
• Provide documents as context to your assistant with RAG