Agent stops responding after initial greeting in Gateway setup

[Karthikeyareddy183]

Describe the bug
The LiveKit Agent, when integrated with a custom FastAPI gateway (server.py) and a Python audio client (client.py), performs its initial greeting successfully. However, after this initial greeting, the agent ceases to respond to any subsequent audio input from the client. No further audio is played back through the client's speaker.

Server

• Version: [uvicorn server:app --host 0.0.0.0 --port 8000] (This is the web server hosting FastAPI)
• Environment: local dev
• Operating System: Windows 10/11 (or your specific OS)
• Python Version: Python 3.x.x (e.g., 3.9.12)
• FastAPI Version: [Your FastAPI version]
• livekit-rtc Version: 1.0.11
• opuslib Version: 3.0.1
• uvicorn Version: 0.34.0
• LiveKit Cloud URL: wss://esp32-zlnycimc.livekit.cloud/rtc
• LiveKit Room Name: test-room

Client

• SDK: Python (custom client using websockets and sounddevice)
• Version: N/A (Custom script, not an SDK version)
• Client Details:
  - Operating System: Windows 10/11 (or your specific OS)
  - Python Version: 3.11.0
  - websockets Version: 15.0.1
  - sounddevice Version: 0.5.2
  - numpy Version: 1.26.4
  - Audio SAMPLE_RATE: 48000 Hz
  - Audio SAMPLES_PER_FRAME: 960

To Reproduce
Steps to reproduce the behavior:

1. Ensure config.py is properly configured with your LiveKit credentials (LIVEKIT_WS_URL, LK_API_KEY, LK_API_SECRET) and ROOM_NAME.
2. Start the gateway server: uvicorn server:app --host 0.0.0.0 --port 8000
3. In a separate terminal, start the LiveKit agent in development mode: python agent.py dev
4. In another separate terminal, start the audio client: python client.py
5. Observe the initial audio greeting from the agent played back through the client's speaker.
6. Attempt to speak to the agent via the client's microphone.
7. See error: The agent does not provide any further audio responses.

Expected behavior
The LiveKit agent should maintain a continuous, bidirectional voice conversation. After the initial greeting, it should process subsequent audio input from the client's microphone and stream its generated responses back to the client's speaker.

Additional context
Relevant Logs:

--- server.py logs (during client connection and interaction) ---
[Paste full server.py logs from when you start uvicorn, the client connects, and you try to speak]
Example relevant lines:
INFO:      ('127.0.0.1', 53617) - "WebSocket /esp32" [accepted]
✅ ESP32 connected
INFO:      connection open
livekit_api::signal_client:159:livekit_api::signal_client - unexpected signal error: ws failure: HTTP error: 200 OK
→ RTC connected to LiveKit
Waiting for remote audio track...
Participant connected: agent-AJ_EpDKUyxqxHHU
[Include any further logs, especially if "Found remote audio track..." or "[Gateway:Rx] Received audio frame..." appear for the agent's initial greeting, and then stop appearing.]

--- client.py logs (during connection and interaction) ---
[Paste full client.py logs from when you run the script]
Example relevant lines:
2025-07-16 11:55:06,671 [INFO] 🎧 In stream ready
2025-07-16 11:55:06,817 [INFO] 🎧 Out stream ready
2025-07-16 11:55:06,817 [INFO] 🔗 Connecting …
2025-07-16 11:55:08,916 [INFO] ✅ Connected to server

python test.py dev
2025-07-17 15:38:02,741 - DEBUG asyncio - Using proactor: IocpProactor 
2025-07-17 15:38:02,746 - DEV  livekit.agents - Watching D:\Livekit with esp32\agent 
2025-07-17 15:38:05,345 - DEBUG asyncio - Using proactor: IocpProactor 
2025-07-17 15:38:05,355 - INFO livekit.agents - starting worker {"version": "1.1.7", "rtc-version": "1.0.11"}
2025-07-17 15:38:05,355 - INFO livekit.agents - starting inference executor 
2025-07-17 15:38:05,374 - INFO livekit.agents - initializing process {"pid": 18936, "inference": true}
2025-07-17 15:38:08,002 - DEBUG livekit.agents - initializing inference runner {"runner": "lk_end_of_utterance_multilingual", "pid": 18936, "inference": true}
2025-07-17 15:38:20,942 - DEBUG livekit.agents - inference runner initialized {"runner": "lk_end_of_utterance_multilingual", "elapsed_time": 12.940458600001875, "pid": 18936, "inference": true}
2025-07-17 15:38:20,944 - INFO livekit.agents - process initialized {"pid": 18936, "inference": true, "elapsed_time": 15.57}
2025-07-17 15:38:20,943 - DEBUG asyncio - Using proactor: IocpProactor {"pid": 18936, "inference": true}
2025-07-17 15:38:20,949 - INFO livekit.agents - see tracing information at http://localhost:62770/debug
2025-07-17 15:38:21,252 - INFO livekit.agents - registered worker {"id": "AW_eP3v3h2c8hpD", "url": "wss://esp32-zlnycimc.livekit.cloud", "region": "India", "protocol": 16}
2025-07-17 15:38:28,418 - INFO livekit.agents - received job request {"job_id": "AJ_aCUycoMzRtod", "dispatch_id": "", "room_name": "test-room", "agent_name": "", "resuming": false}
2025-07-17 15:38:28,475 - INFO livekit.agents - initializing job runner {"tid": 24000}
2025-07-17 15:38:28,475 - DEBUG asyncio - Using proactor: IocpProactor      
2025-07-17 15:38:28,476 - INFO livekit.agents - job runner initialized {"tid": 24000, "elapsed_time": 0.0}        
2025-07-17 15:38:30,700 - DEBUG livekit.agents - http_session(): creating a new httpclient ctx

agent.txt
client.txt
config.txt
opus_utils.txt
server.txt

[JATIN7838]

I am facing the same issue, thought there is an error but then implemented :

def on_track_subscribed(track, publication, participant):
print(f"🎧 Agent: Track subscribed from {participant.identity}: {track.sid}")
if participant.identity == "caller-123" and track.kind == TrackKind.KIND_AUDIO:
print(f"🎧 Agent: Successfully subscribed to client audio track!")
print(f"🎯 STT should now work with client audio!")

        # Monitor audio levels from the subscribed track
        stream = AudioStream(track)
        async def monitor_audio_levels():
            frame_count = 0
            async for event in stream:
                frame_count += 1                    
                if frame_count % 100 == 0:
                    audio_level = max(abs(int.from_bytes(event.frame.data[i:i+2], byteorder='little', signed=True)) 
                                    for i in range(0, min(len(event.frame.data), 100), 2))
                    print(f"🎤 Agent received audio: Frame #{frame_count}, level={audio_level}, bytes={len(event.frame.data)}")                        
        asyncio.create_task(monitor_audio_levels())

Output:

🎤 Agent received audio: Frame #5000, level=65537, bytes=480
🎤 Agent received audio: Frame #5100, level=65538, bytes=480
🎤 Agent received audio: Frame #5200, level=131071, bytes=480
🎤 Agent received audio: Frame #5300, level=65536, bytes=480
🎤 Agent received audio: Frame #5400, level=65537, bytes=480

the audio levels too high on mic they wont exceed 1000. Then implemented audio buffer which will save the audio received every 5 seconds to .mp4 or .wav. Turns out the audio files have clear audio like I said in the mic. Was going insane for 5 hours. Hopefully this is resolved.