Skip to content
BranchesTags

Repository files navigation

Azimov Voice

A real-time voice conversation application built with Next.js and OpenAI's Realtime API.

Table of Contents

  1. How It Works
  2. Models & Technology
  3. Architectural Decisions
  4. Scope: Current Build vs. Production
  5. Core Systems
  6. External Integrations

How It Works

Azimov Voice creates real-time, bidirectional voice conversations between users and an AI persona. Here's the end-to-end flow:

  1. Page Load (Server-Side)

    • Vercel's edge network provides user location via IP geolocation headers
    • Recent news/events are pre-fetched from Exa AI to give the agent topical context
    • A random personality is selected from 1000 pre-generated personas (each with detailed backstory, interests, profession, cultural background)

  2. Session Initiation (Client-Side)

    • User clicks the microphone button to start
    • A short-lived ephemeral API key is fetched from the server (keeps main API key secure)
    • WebRTC connection is established directly with OpenAI's Realtime API
    • The AI agent is configured with the personality, location context, and recent events
    • Agent initiates conversation with a greeting in-character

  3. Conversation Loop

    • User's voice is streamed via WebRTC to OpenAI; transcript is streamed back
    • If the agent needs current information, it calls the web_search tool (Exa API)
    • Response audio along with transcript is streamed back via WebRTC.
    • Semantic VAD (Voice Activity Detection) handles turn-taking automatically

  4. Engagement Features

    • Idle timeout: If user is silent for ~1 minute, agent proactively re-engages
    • Multimodal input: Users can also type messages or attach images
    • Visual feedback: Animated orb shows who's speaking (purple = agent, green = user)
┌─────────────────────────────────────────────────────────────────┐
│                         Browser                                 │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐  │
│  │ VoiceChat   │  │ VoiceOrb    │  │ ChatTranscript          │  │
│  │ Component   │  │ (Visual)    │  │ (Conversation History)  │  │
│  └──────┬──────┘  └─────────────┘  └─────────────────────────┘  │
│         │                                                       │
│  ┌──────▼──────────────────────────────────────────────────┐    │
│  │              useVoiceSession Hook                       │    │
│  │  - Session state management                             │    │
│  │  - WebRTC transport handling                            │    │
│  │  - History tracking & timestamps                        │    │
│  └──────┬──────────────────────────────────────────────────┘    │
│         │                                                       │
│  ┌──────▼──────────────────────────────────────────────────┐    │
│  │     OpenAIRealtimeWebRTC + RealtimeSession              │    │
│  │     (@openai/agents/realtime)                           │    │
│  └──────┬─────────────────────────────────────────┬────────┘    │
└─────────┼─────────────────────────────────────────┼─────────────┘
          │                                         │
          │ WebRTC                                  │
          │                                         │
          ▼                                         ▼
┌───────────────────────────────────┐     ┌───────────────────────────────┐
│      OpenAI Realtime API          │     │        Next.js Server         │
│───────────────────────────────────│     │───────────────────────────────│
│  - gpt-realtime                   │     │ - Ephemeral key generation    │
│  - Semantic VAD                   │     │ - Tool execution (web_search) │
└───────────────────────────────────┘     └───────────────┬───────────────┘
                                                          │
                                                          │ Search API
                                                          ▼
                                          ┌───────────────────────┐
                                          │        Exa AI         │
                                          └───────────────────────┘

Models & Tech

AI Models

gpt-realtime which uses whisper-1 for STT.

Voice Options

Voices are selected based on the persona's gender:

  • Male voices: ash, cedar, echo, verse
  • Female voices: alloy, ballad, coral, marin, sage, shimmer

Tech Stack

  • Next.js
  • TypeScript
  • OpenAI Agents SDK
  • Tailwind
  • shadcn-ui
  • Vercel AI Elements
  • Zod

Architectural Decisions

1. WebRTC

Decision: Use OpenAIRealtimeWebRTC transport instead of WebSocket.

Why:

  • Lower latency for audio streaming (peer-to-peer)
  • Better handling of real-time bidirectional audio
  • Built-in echo cancellation and noise reduction

Alternatives considered:

  • WebSocket transport

    WebRTC provides lower latency and handles audio input and output automatically.

2. Ephemeral Key Authentication

Decision: Server generates short-lived client tokens; client never sees main API key.

Why:

  • Security: API key never exposed to browser
  • Token rotation: Each session gets a fresh, limited-scope token
  • Standard pattern for client-side API access

Alternatives considered:

  • Proxying all requests through server

    Much lower latency for direct WebRTC connection.

3. Pre-generated Personality Dataset

Decision: 1000 personalities stored in a static JSON file, randomly assigned per session.

Why:

  • Consistent, high-quality personas with rich backstories

  • Uses Nvidia's Nemotron-Personas-USA dataset with an extraction script.

    Dataset contains 1M personas so this can always be expanded.

4. Server-Side Context Fetching

Decision: Location and recent events fetched in root layout (server component).

Why:

  • Data available before hydration
  • No loading states for initial context
  • Leverages Next.js server components

Alternatives considered:

  • Client-side fetching with loading states

    • Would need to send more data to the client than necessary for the personalities.
    • Since the web search is executed from the server, it will add an unnecessary round trip.

5. React Context for Cross-Cutting Data

Decision: Location, RecentEvents, and Personality passed via React Context.

Why:

  • Avoids prop drilling through component tree
  • Data set once at root, consumed anywhere
  • Clean provider pattern

Scope: Current Build vs. Production

What Was Built (Current Prototype)

  • Real-time voice conversation
  • Dynamic AI personalities (1000)
  • Location-aware context
  • Recent events context
  • Web search
  • Text message fallback
  • Image attachment support
  • Pause/resume session
  • Idle re-engagement
  • Response latency display

What a Production Version Would Include

  • Long-term Memory
  • Authentication
  • Conversation Persistence
  • Better UI
  • Personality Selection
  • Rate Limiting
  • Analytics
  • Error Tracking
  • Accessibility
  • Choosing Voice

Architecture

Voice Session Management

The useVoiceSession hook (hooks/use-voice-session.ts) is the central state manager for voice interactions:

type SessionState = "idle" | "connecting" | "connected" | "paused" | "error"

// Hook exports
{
  state: SessionState,          // Current connection state
  history: RealtimeItem[],      // Conversation history
  timestamps: MessageTimestamps, // Message timing data
  submittedImages: SubmittedImages, // Attached images
  isAgentTurn: boolean,         // Is AI currently speaking
  isUserTurn: boolean,          // Is user expected to speak
  isPaused: boolean,            // Session paused state
  
  // Actions
  connect: () => Promise<void>, // Start a new session
  pause: () => void,            // Pause (mute) session
  resume: () => void,           // Resume session
  newChat: () => Promise<void>, // Clear and restart
  clear: () => void,            // End session completely
  interrupt: () => void,        // Interrupt AI response
  sendMessage: (text: string) => void,    // Send text message
  addImage: (dataUrl: string, trigger?: boolean) => void, // Add image
  storeSubmittedImages: (urls: string[]) => void, // Track images
}

Idle Engagement: After ~1 minute of silence, a system message is injected and the agent attempts to re-engage with a new topic.

Latency Tracking: Messages are timestamped when received and when assistant starts responding. Latency = assistant_started - user_timestamp.

Image Handling: Files added via input/paste/drag-drop → blob URLs for preview → converted to data URLs on submit → sent via session.addImage().

AI Agent Configuration

The createVoiceAgent function (lib/agent.ts) constructs a RealtimeAgent with:

function createVoiceAgent(
  city: string | null,
  country: string | null,
  recentEvents: string | null,
  personality: { prompt: string; sex: string } | null,
  voice: string
): RealtimeAgent

System Instructions Include:

  • Personality prompt (if assigned)
  • Location and current datetime
  • Conversational guidelines (emotive tone, small talk first)
  • Turn-taking behavior (respond on cues, not every pause)
  • Language handling (match user's language, default English)
  • Recent events context for topical conversation

Tools:

  • web_search: Real-time web search via Exa API

Context Providers

Three React Context providers supply data to client components:

LocationProvider (lib/location-context.tsx): Provides user's geographic location from Vercel headers.

RecentEventsProvider (lib/recent-events-context.tsx): Pre-fetches recent news for conversation topics.

PersonalityProvider (lib/personality-context.tsx): Randomly assigned personality from 1000 profiles. Each personality includes demographics, professional/cultural background, skills, hobbies, and career goals.

Session Events

Helper functions in lib/session-events.ts:

function sendStartConversation(session: RealtimeSession): void
function sendIdleEngagementPrompt(session: RealtimeSession): void
function deriveTurnState(history: RealtimeItem[], isConnected: boolean): { lastMessage, isUserTurn, isAgentTurn }
function getLastMessageId(history: RealtimeItem[]): string | null

External Integrations

OpenAI Realtime API

Configuration (lib/session-config.ts):

{
  audio: {
    input: {
      format: "pcm16",
      noiseReduction: { type: "near_field" },
      transcription: { model: "whisper-1" },
      turnDetection: {
        type: "semantic_vad",  // Smart pause detection
        createResponse: true,  // Auto-respond on turn end
        interruptResponse: true, // Allow interruptions
        eagerness: "medium",
      },
    },
    output: { format: "pcm16" },
  },
}

Environment: OPENAI_API_KEY

Exa AI Search

Server Action (lib/actions.ts):

async function searchWebWithExa({
  query: string,
  country: string | null,
}): Promise<string>
  • Uses Exa's /search endpoint with contents parameter
  • Returns contextualized search results (max 10,000 chars)
  • User location passed for relevance

Environment: EXA_API_KEY

About

azimov-voice.vercel.app

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages