livekit-agents@1.6.0

Introducing Asynchronous Tools

When a long-running tool is in progress, the user will be met with silence until it completes. Asynchronous tools can hand control back to the LLM before it finishes, streaming updates into the conversation as it progresses.

class TravelAgent(Agent):
    @llm.function_tool(flags=llm.ToolFlag.CANCELLABLE, on_duplicate="confirm")
    async def book_flight(self, ctx: RunContext, origin: str, destination: str, date: str) -> str:
        """Book a flight."""
        # First update is delivered immediately and releases control back to the LLM,
        # so the agent can say e.g. "Sure, searching flights to Tokyo — this'll take a minute."
        await ctx.update(
            f"Searching flights from {origin} to {destination} on {date}. "
            "This will take a couple of minutes."
        )

        await asyncio.sleep(30)  # simulate long-running work

        # Later updates are coalesced into a deferred reply, delivered when the agent
        # is idle: "Good news — best price is $289 on Delta, confirming now."
        await ctx.update("Found 3 options. Best price $289 on Delta. Confirming now.")

        await asyncio.sleep(40)  # simulate more long-running work

        # The final return is also delivered when the agent is idle.
        return f"Booked! Confirmation FL-{random.randint(100000, 999999)}."

Filler phrases

Use filler words to break long silences via ctx.with_filler(). You can set delay to the amount of quiet seconds per interval for when the filler words are played. You can also rotate between phrases, set interval to configure the cooldown between filler fires.

        followups = [
            "Almost there, just confirming.",
            "Still working on it, won't be long.",
            "Hang tight — almost done.",
        ]
        async with ctx.with_filler(
            lambda step: followups[step], delay=5, interval=10, max_steps=len(followups)
        ):
            await asyncio.sleep(40) # simulate a long API call

Read more about async tools in our docs.

Complete Changelog

• test: categorize tests and select via pytest --unit/--plugin openai/ … flags by @Bobronium in #5945
• hamming plugin: add customer_metadata, external_links, and session_mode fields by @sumanyu in #5263
• fix(job): log unhandled exceptions in fire-and-forget tasks by @theomonnom in #5961
• ci(deploy-examples): drop manifest publishing job by @theomonnom in #5963
• ci(deploy-examples): deploy from the dispatched branch instead of hardcoded main by @theomonnom in #5964
• feat(voice): flush audio and text output per FlushSentinel by @longcw in #5956
• add ctx.with_filler() for acoustic fillers during long tools by @longcw in #5885
• first-class async tools via ctx.update() by @longcw in #5841
• Deploy examples against the deployed git ref instead of PyPI by @theomonnom in #5969
• feat(job): expose single_peer_connection in JobContext.connect by @renal128 in #5966
• docs: add filler_agent.py to examples README table of contents by @detail-app[bot] in #5967
• docs: update test commands to use pytest category flags by @detail-app[bot] in #5958
• Enable git-lfs in example images so git-installed deps get real LFS binaries by @theomonnom in #5970
• fix(ipc): rate-limit and contextualize high-memory warnings by @Bobronium in #5962
• (examples deploy): fix lfs by @tinalenguyen in #5973
• (openai responses): surface phase field by @tinalenguyen in #5975
• fix(openai): skip empty messages in realtime chat updates by @nightcityblade in #5952
• fix quick reconnect participant keyerror by @tinalenguyen in #5979
• livekit-agents@1.5.18 by @github-actions[bot] in #5981
• fix(google): require compatible google-genai version by @nightcityblade in #5988
• Hotel Receptionist example by @theomonnom in #5983
• fix(openai): handle string realtime status details by @he-yufeng in #5994
• fix(google): preserve request timeout with http options by @he-yufeng in #5976
• fix: resolve shutdown futures when IPC reader stops by @longcw in #6003
• livekit-agents@1.5.19.rc1 by @github-actions[bot] in #5986
• simulation glue code by @theomonnom in #5688
• feat(cli): python -m livekit.agents start|console discovers the AgentServer by @theomonnom in #6024
• tests: speed up unit tests by @Bobronium in #5980
• feat(bargein): add default threshold support and drop http transport by @chenghao-mou in #5946
• feat(anam): forward output video dimensions via SessionOptions by @Smidge in #5935
• ci: speedup type checking by @Bobronium in #5982
• simulation: shut the job down when the simulator disconnects by @theomonnom in #6037
• fix(sarvam): emit speech timing for STT metrics by @dhruvladia-sarvam in #5995
• fix(elevenlabs): end server vad turns by @he-yufeng in #5872
• (inference tts): add inworld delivery_mode param by @tinalenguyen in #6048
• fix(aws): recover Nova Sonic system instability errors by @nightcityblade in #6042
• simulation: --simulation flag disables the worker load limit; text sims skip STT/TTS by @theomonnom in #6036
• fix(voice): route and drain AsyncToolset executors correctly on handoff by @toubatbrian in #6049
• fix(deepgram): use stored language when validating model in update_options by @adityajha2005 in #6041
• simulation: read the dispatch from a simulator participant attribute by @theomonnom in #6053
• remove the next-release changeset machinery by @theomonnom in #6054
• fix(bargein): error when no interruption threshold is known by @chenghao-mou in #6034
• feat(assemblyai): add agent_context, previous_context_n_turns, and u3-rt-pro-beta-1 streaming options by @dlange-aai in #6017
• simulation: don't start the worker HTTP server by @theomonnom in #6055
• simulation: don't cache a pre-connect simulation_context miss by @theomonnom in #6061
• (sarvam stt): fix end of speech timing by @tinalenguyen in #6052
• feat(elevenlabs): add no_verbatim option to STT by @jaime-wiselook in #6032
• fix(google): correct pitch parameter type from int to float in TTS by @adityajha2005 in #6058
• (deepgram stt): add keyterm to batch config by @tinalenguyen in #6062
• Elevenlabs stream api missing flags by @devb-enp in #6031
• restore dev mode log format by @theomonnom in #6065
• fix(neuphonic): use configured encoding and lang_code in TTS requests by @adityajha2005 in #6064
• livekit-agents@1.6.0 by @github-actions[bot] in #6066

New Contributors

• @Bobronium made their first contribution in #5945
• @sumanyu made their first contribution in #5263
• @renal128 made their first contribution in #5966
• @Smidge made their first contribution in #5935
• @adityajha2005 made their first contribution in #6041
• @jaime-wiselook made their first contribution in #6032

Full Changelog: https://github.com/livekit/agents/compare/livekit-agents@1.5.17...livekit-agents@1.6.0