feat(agents): add Toolset support to ToolContext and AgentActivity

[toubatbrian]

Summary

Adds Python-parity Toolset support: a stateful container that bundles a group of tools behind shared setup() / aclose() lifecycle hooks. Toolset instances can be passed directly into Agent({ tools: [...] }) alongside individual function tools; their tools are flattened into the agent's ToolContext and the runtime drives setup on activity start, close on activity teardown, and a diff on updateTools().

Closes the placeholder TODOs left in tool_context.ts from the previous list-syntax refactor.

Changes

Core (agents/src/llm/tool_context.ts)

• New Toolset class with id, tools, setup(), aclose() (default no-ops, override in subclass).
• New ToolCalledEvent / ToolCompletedEvent payload types (exported for plugin authors).
• ToolContextEntry widened to FunctionTool | ProviderDefinedTool | Toolset.
• updateTools() recurses into Toolsets and flattens their tools into _functionToolsMap / _providerTools while tracking the Toolset itself in _toolsets.
• equals() compares _toolsets as an identity set (matches Python's {id(ts) for ts in self._tool_sets} semantics — order-insensitive).
• Duplicate function-tool names continue to throw duplicate function name: <name>, including when the duplicate is contributed by a Toolset (Python parity).

Activity (agents/src/voice/agent_activity.ts)

• New setupToolsets() / closeToolsets() driven by _startSession and _closeSessionResources.
• setup() failures propagate to the caller after rolling back any toolsets that already initialized — the agent fails explicitly rather than running with half-set-up resources.
• updateTools() now diffs added/removed Toolsets and runs lifecycle in setup → swap toolCtx → close order so a setup failure leaves the activity pointing at the previous (still-valid) ToolContext.
• The IGNORE_ON_ENTER filter recurses through Toolsets so nested function tools also honor the flag.

Plugin migrations to ToolContext.flatten() + isFunctionTool filter
Toolset-contributed tools are included in _functionToolsMap, but several plugins were also rebuilding tool lists from Object.entries(toolCtx.functionTools), which would silently drop the Toolset references and confuse subsequent updateTools() diffs. Migrated:

• plugins/google/src/utils.ts
• plugins/mistralai/src/llm.ts
• plugins/openai/src/realtime/realtime_model.ts — also fixes a real bug in _handleSessionUpdated() where the post-update retainedTools rebuild dropped Toolsets, causing them to be diffed-as-removed and prematurely aclose()d.
• plugins/openai/src/responses/llm.ts
• plugins/openai/src/ws/llm.ts
• plugins/phonic/src/realtime/realtime_model.ts

plugins/openai/src/llm.ts (chat) and plugins/baseten/src/llm.ts left unchanged: they iterate Object.keys(toolCtx.functionTools) which already picks up flattened Toolset tools, mirroring Python's chat plugin.

Tests

• agents/src/llm/tool_context.test.ts: new Toolset describe block (id/tools accessors, default no-op lifecycle, subclass override, flatten-into-ToolContext, duplicate-throw parity, identity-set equals() parity).
• plugins/test/src/llm.ts: new toolset describe block runs against every LLM plugin's shared test suite — verifies the model can call a function tool nested inside a Toolset and that direct tools continue to work alongside.
• agent_activity.test.ts / agent_activity_handoff.test.ts: existing tests pass (verified the toolset wiring doesn't disturb the fake activity scaffolding).

Example

• New examples/src/basic_toolsets.ts showing a location_tools Toolset bundling getWeather + lookupTimezone and passed alongside an Agent.

Test plan

• pnpm test --run agents/src/llm/tool_context.test.ts — 46/46 pass (40 prior + 6 new)
• pnpm test --run agents/src/voice/agent_activity.test.ts agents/src/voice/agent_activity_handoff.test.ts — 24/24 pass
• pnpm test --run plugins/openai/src/llm.test.ts plugins/google/src/llm.test.ts plugins/mistralai/src/llm.test.ts — 29/30 pass (1 pre-existing skip), including 2 new toolset cases per plugin
• pnpm build — all 32 packages green
• pnpm lint — clean (only pre-existing any warnings in unrelated files)
• Manual: verify toolset aclose() runs on session close and that setup() rejection rolls back already-set-up toolsets

[changeset-bot]

🦋 Changeset detected

Latest commit: 4c3794f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 31 packages

Name	Type
@livekit/agents	Major
@livekit/agents-plugin-google	Major
@livekit/agents-plugin-openai	Major
@livekit/agents-plugin-anam	Major
@livekit/agents-plugin-assemblyai	Major
@livekit/agents-plugin-baseten	Major
@livekit/agents-plugin-bey	Major
@livekit/agents-plugin-cartesia	Major
@livekit/agents-plugin-cerebras	Major
@livekit/agents-plugin-deepgram	Major
@livekit/agents-plugin-elevenlabs	Major
@livekit/agents-plugin-fishaudio	Major
@livekit/agents-plugin-hedra	Major
@livekit/agents-plugin-hume	Major
@livekit/agents-plugin-inworld	Major
@livekit/agents-plugin-lemonslice	Major
@livekit/agents-plugin-liveavatar	Major
@livekit/agents-plugin-livekit	Major
@livekit/agents-plugin-minimax	Major
@livekit/agents-plugin-mistral	Major
@livekit/agents-plugin-mistralai	Major
@livekit/agents-plugin-neuphonic	Major
@livekit/agents-plugin-phonic	Major
@livekit/agents-plugin-resemble	Major
@livekit/agents-plugin-rime	Major
@livekit/agents-plugin-runway	Major
@livekit/agents-plugin-sarvam	Major
@livekit/agents-plugin-silero	Major
@livekit/agents-plugins-test	Major
@livekit/agents-plugin-trugen	Major
@livekit/agents-plugin-xai	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ toubatbrian
❌ claude
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 15 additional findings in Devin Review.

[devin-ai-integration]

🔴 Google toToolsConfig incorrectly throws when function tools coexist with provider tools or geminiTools

The new toToolsConfig helper throws "Gemini does not support mixing function tools and provider tools" whenever function declarations and provider-side tools exist simultaneously (plugins/google/src/utils.ts:204-205). This is a regression: the old realtime code (buildConnectConfig) merged functionDeclarations and geminiTools (e.g. { codeExecution: {} }) into a single types.Tool entry, which worked. The Gemini API accepts multiple Tool entries in the array — one for function declarations and separate ones for googleSearch, codeExecution, etc. — so rejecting the combination outright is incorrect.

This breaks two paths:

1. Realtime sessions — any user passing geminiTools (e.g. { codeExecution: {} }) alongside agents that have function tools will now get a runtime error instead of the previously-working merged behavior.
2. New GeminiTool provider tools — the newly-introduced GoogleSearch, ToolCodeExecution, etc. cannot be used together with function tools, which is a core use case for provider tools.

The onlySingleType flag used by the non-realtime LLM path (plugins/google/src/llm.ts:361) also cannot help since the throw fires before that check.

Prompt for agents

The error at plugins/google/src/utils.ts:204-206 incorrectly prevents valid Gemini API configurations where function declarations coexist with other tool types (googleSearch, codeExecution, etc.).

The Gemini API accepts a Tool[] array where function declarations and other tool types are separate entries. The fix should remove the throw and instead allow both types to coexist as separate entries in the returned array.

However, care is needed for the onlySingleType=true path (used by the non-realtime LLM at plugins/google/src/llm.ts:361). When onlySingleType is true and both function tools and provider tools exist, the current early-return at line 208 would silently drop provider tools. The fix should either:
1. Remove onlySingleType entirely and always merge both types into the array (function declarations in one entry, provider tools in separate entries), OR
2. When onlySingleType is true and both types exist, merge them into a single types.Tool object (like the old realtime code did with spread), OR
3. Keep onlySingleType but ensure provider tools are not silently dropped — perhaps by merging them into the same Tool entry that holds functionDeclarations.

The old realtime code used spread to merge: { functionDeclarations: [...], ...geminiTools } which put everything in one Tool. The new architecture with separate GeminiTool subclasses (each producing their own toToolConfig()) naturally maps to separate Tool[] entries. Both approaches are valid per the Gemini API.

Affected call sites: plugins/google/src/llm.ts:358-362 (LLM path) and plugins/google/src/beta/realtime/realtime_api.ts:1416-1420 (realtime path).

---

Was this helpful? React with 👍 or 👎 to provide feedback.