Python: [Bug]: Gemini chat client can't call the harness tool set

[antsok]

Description

Selecting a Google model and starting a run of a harness agent (create_harness_agent) over GeminiChatClient crashes the stream during request preparation — before or at the first model call — for two independent, Gemini-specific reasons. The same agent and tool set run unchanged on Ollama, the OpenAI-compatible clients, and Azure Foundry.

1. Tool parameter schemas. _prepare_tools builds each types.FunctionDeclaration with parameters=tool.parameters(). That field is a google.genai types.Schema (an OpenAPI-3.0 subset) that rejects the JSON-Schema keywords the harness's own tools emit — $ref/$defs (the built-in todos_add(todos: list[_TodoAddItemSchema]) tool) and oneOf (any tool with an optional/union object argument). Pydantic raises ValidationError at declaration-build time, so no request is ever sent.

2. Built-in web search mixed with function calling. The harness adds a web-search tool by default (create_harness_agent(disable_web_search=False)); for a client that supports it, that becomes client.get_web_search_tool(). On Gemini that is a native types.Tool(google_search=...) built-in. Gemini rejects a request that mixes a built-in tool with function declarations unless tool_config.include_server_side_tool_invocations is enabled — which the client never sets.

Either defect alone is fatal; 1 fires first, so 2 is only reachable after 1 is fixed. Both are Gemini-only: Ollama doesn't implement SupportsWebSearchTool, and the OpenAI-compatible/Foundry clients accept $ref/$defs/oneOf schemas.

Verified against: upstream main @ 0260ea0e (2026-07-03); installed agent-framework-gemini 1.0.0a260630, agent-framework-core 1.10.0, google-genai 1.75.0, pydantic 2.13.4
Provider under test: Gemini Developer API (Google AI Studio), api_key auth, model gemini-3.5-flash (also reproduces on gemini-3.1-flash-lite; independent of model)

2. Symptoms

User-visible (any AG-UI frontend; observed with CopilotKit): the user picks a Google model, sends a message, and the reply is a generic "An internal error has occurred while streaming events." No tokens ever stream.

Backend traceback (defect #1), captured via OpenTelemetry logs:

File ".../agent_framework_gemini/_chat_client.py", line 915, in _prepare_tools
    types.FunctionDeclaration(
pydantic_core._pydantic_core.ValidationError: 2 validation errors for FunctionDeclaration
parameters.properties.todos.items.$ref
  Extra inputs are not permitted [type=extra_forbidden, input_value='#/$defs/_TodoAddItemSchema', ...]
parameters.$defs
  Extra inputs are not permitted [type=extra_forbidden, input_value={'_TodoAddItemSchema': {...}}, ...]

A tool with an optional object argument fails the same way on parameters.properties.<arg>.oneOf.

Backend traceback (defect 2, once 1 is fixed):

File ".../agent_framework_gemini/_chat_client.py", line 541, in _stream
    async for chunk in await self._genai_client.aio.models.generate_content_stream(...)
google.genai.errors.ClientError: 400 Bad Request. {
  "error": { "code": 400,
    "message": "Please enable tool_config.include_server_side_tool_invocations to use Built-in tools with Function calling.",
    "status": "INVALID_ARGUMENT" } }

3. Reproduction

Both are reproducible in isolation against the live Gemini Developer API — no framework harness required.

3.1 Defect 1 — tool schema ($ref/$defs, oneOf)

from typing import TypedDict
from typing_extensions import NotRequired
from agent_framework import tool
from google.genai import types

class _TodoAddItemSchema(TypedDict):   # must be module-scope for the forward ref
    title: str
    description: NotRequired[str]

@tool(name="todos_add")
async def todos_add(todos: list[_TodoAddItemSchema]) -> str:
    "Add todos."
    return "ok"

schema = todos_add.parameters()        # -> {"$defs": {...}, "properties": {"todos": {"items": {"$ref": "#/$defs/_TodoAddItemSchema"}, ...}}, ...}
types.FunctionDeclaration(name="todos_add", description="x", parameters=schema)   # ValidationError: $ref / $defs "Extra inputs are not permitted"

# oneOf variant (optional free-form object arg) fails the same way:
oneof = {"type": "object", "properties": {"args": {"oneOf": [{"type": "object"}, {"type": "null"}]}}}
types.FunctionDeclaration(name="t", description="x", parameters=oneof)            # ValidationError: parameters.properties.args.oneOf

This is exactly what create_harness_agent triggers: its TodoProvider.todos_add tool produces the $ref/$defs schema above.

3.2 Defect 2 — built-in tool + function calling

import asyncio
from google import genai
from google.genai import types

client = genai.Client(api_key="<GEMINI_API_KEY>")   # Developer API / Google AI Studio
fn = types.Tool(function_declarations=[types.FunctionDeclaration(
    name="get_weather", description="Get weather",
    parameters_json_schema={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]})])
search = client and types.Tool(google_search=types.GoogleSearch())   # what the harness auto-adds on Gemini
auto = types.FunctionCallingConfig(mode="AUTO")

async def go(tool_config):
    stream = await client.aio.models.generate_content_stream(
        model="gemini-3.5-flash", contents="What's the weather in Paris?",
        config=types.GenerateContentConfig(tools=[search, fn], tool_config=tool_config))
    async for _ in stream: pass

asyncio.run(go(types.ToolConfig(function_calling_config=auto)))                                       # 400 INVALID_ARGUMENT (as above)
asyncio.run(go(types.ToolConfig(function_calling_config=auto, include_server_side_tool_invocations=True)))  # OK

End-to-end: run create_harness_agent(..., disable_web_search=False) (the default) over GeminiChatClient with any function tools and send any prompt.

Code Sample

Error Messages / Stack Traces

Package Versions

agent-framework-gemini: 1.0.0a260630, agent-framework-core: 1.10.0

Python Version

Python 3.12

Additional Context

4. Root cause

4.1 Defect 1 — _prepare_tools passes raw JSON Schema to the OpenAPI-subset parameters

RawGeminiChatClient._prepare_tools (installed build _chat_client.py:883) builds declarations directly from the tool's JSON schema:

types.FunctionDeclaration(
    name=tool.name,
    description=tool.description or "",
    parameters=tool.parameters(),   # line ~915: raw JSON Schema -> types.Schema
)

types.FunctionDeclaration.parameters is typed types.Schema, the OpenAPI-3.0 subset, which disallows $ref, $defs, definitions, and oneOf (only anyOf is modelled, and unions must be inlined). Pydantic rejects the extra keys with extra_forbidden. Tools whose parameters reference a nested model (Pydantic/TypedDict → $ref + $defs) or carry a union (oneOf) therefore can never be declared. This is a well-known Gemini-API constraint that peer SDKs hit and solve by flattening or by using JSON-Schema passthrough (see §6).

flowchart LR
    A["tool.parameters()<br/>JSON Schema w/ $ref/$defs or oneOf"] --> B["types.FunctionDeclaration(parameters=...)"]
    B --> C{"types.Schema<br/>(OpenAPI-3.0 subset)"}
    C -->|"$ref / $defs / oneOf"| D["pydantic ValidationError<br/>extra_forbidden"]
    style D fill:#fdd,stroke:#c00

Loading

4.2 Defect 2 — the harness adds a built-in tool the client never flags for

create_harness_agent (core _harness/_agent.py, disable_web_search: bool = False by default) appends a web-search tool when the client supports it:

if not disable_web_search:
    if isinstance(client, SupportsWebSearchTool):
        assembled_tools.append(client.get_web_search_tool())   # Gemini -> types.Tool(google_search=...)
    else:
        logger.warning("Web search tool not available: client %r does not implement SupportsWebSearchTool. ...")

GeminiChatClient implements SupportsWebSearchTool, so a native google_search built-in is added and passed through _prepare_tools unchanged, alongside the function declarations. On the Gemini Developer API, mixing a built-in tool with function declarations requires tool_config.include_server_side_tool_invocations=True; the client's _prepare_config/_prepare_tool_config never set it, so the API returns 400. (Ollama's client is not SupportsWebSearchTool, so no built-in is added — hence the crash is Gemini-only.)

Platform nuance (confirmed by peer reports, §6): Vertex AI allows the combination without the flag and rejects the field as unknown, so the flag must be set only on the Developer-API path (the client already tracks this as self._vertexai).

5. Proposed upstream fix

Both fixes live in agent-framework-gemini; neither requires changes to the harness or other providers.

1. Declare tools with parameters_json_schema instead of parameters. google-genai (≥ the pinned 1.75.0) exposes FunctionDeclaration.parameters_json_schema, forwarded to the API as parametersJsonSchema, which the model interprets as full JSON Schema ($ref/$defs, oneOf/anyOf, nullable unions). This offloads the whole compatibility surface to the API rather than lossily transcribing into types.Schema by hand. Exactly one of parameters / parameters_json_schema may be set. (Google's own guidance for complex schemas is to use parametersJsonSchema.) A stricter alternative — flatten $ref/$defs and map oneOf→anyOf into types.Schema — is more code and still lossy for constructs the subset can't express.

2. Enable include_server_side_tool_invocations when a built-in tool is present. In _prepare_config (after building tools), when any entry is a native types.Tool and the client is not in Vertex mode (self._vertexai), set tool_config.include_server_side_tool_invocations = True (creating the ToolConfig if absent). This is Google's prescribed fix and preserves web-search grounding (a documented, intended capability — the framework's own surfaces render web_search/search_tool_result content).

Optional hardening: _prepare_tools could dedupe identical native built-ins (the harness path can present two google_search tools); the API tolerates the duplicate with the flag on, so this is cosmetic.

6. Prior art / duplicate check (2026-07-07)

No existing microsoft/agent-framework issue reports either defect. Adjacent MAF issues found (different failure modes): #3434 (Python, Gemini streaming text lost with usage — closed), #3390 (.NET, Gemini image upload 400), #5490 (.NET tool calling), #6467 (.NET structured-output vs tool calls).

The underlying Gemini-API behaviors are corroborated across the ecosystem, which both confirms the diagnosis and points at the accepted fixes:

Ref	Relevance
google-gemini/gemini-cli#13326, #13142	Gemini API rejects MCP tool schemas using $defs/$ref; fix is schema flattening or JSON-Schema passthrough — same as defect 1
google/adk-python#3995	Google's own ADK mis-parses $defs/$ref — the constraint is API-side, not framework-specific
pydantic/pydantic-ai#1598	Recursive $refs unsupported by Gemini — motivates the cycle fallback in any flatten-based fix
vercel/ai#14369	Gemini rejects $ref/$defs (400 INVALID_ARGUMENT) — same error class as defect 1
vercel/ai#13911	Combining provider-defined (built-in) tools with function tools on Gemini 3: Google AI Studio needs include_server_side_tool_invocations=true; Vertex AI works without it and errors if set — directly validates defect 2's fix and the Vertex gating

7. Verification

• Defect 1: the §3.1 repro raises ValidationError on parameters=; the identical schema on parameters_json_schema= builds cleanly and is accepted by the live API (generate_content_stream streams) for both the $ref/$defs and oneOf shapes.
• Defect 2: the §3.2 repro returns the 400 without the flag and succeeds with it, against gemini-3.5-flash on the Developer API (also with a duplicated google_search tool).
• End-to-end: with the companion app-side shim applied (both fixes), a harness agent on gemini-3.5-flash over AG-UI/CopilotKit completed a full turn — todos_add and other tools executed, an artifact was saved, and the plan streamed — with zero streaming errors and clean telemetry (0 ValidationError, 0 400s). Full app test suite: 128 passed, including 10 new tests covering both crash shapes and both fixes.

8. Suggested issue framing

• Primary issue: "Gemini chat client can't declare tools whose parameters use $ref/$defs/oneOf (built-in todos_add crashes every harness agent)" — defect 1; recommend parameters_json_schema.
• Separable second issue: "Gemini rejects harness web-search built-in mixed with function calls (missing include_server_side_tool_invocations)" — defect 2; note the Developer-API-vs-Vertex gating.

[github-actions]

🤖 Automated triage reproduction notes (agent-authored — trust but verify)

Agent analysis

Repro: RawGeminiChatClient._prepare_tools() in python/packages/gemini/agent_framework_gemini/_chat_client.py:915 uses parameters=tool.parameters() which passes raw JSON Schema ($ref/$defs/oneOf) to types.FunctionDeclaration's OpenAPI-3.0-only parameters field, raising ValidationError. Fix target: use parameters_json_schema= instead. Secondary: _prepare_tool_config (line 928) and _prepare_config (line 830-833) never set include_server_side_tool_invocations=True when built-in tools (google_search) are mixed with function declarations; must be set on Developer API path only (not Vertex). Minimal repro: construct a FakeFunctionTool with $ref/$defs schema, call client._prepare_tools({"tools": [tool]}).

• Failing test: python/packages/gemini/tests/test_issue_6954_repro.py
• Files examined: python/packages/gemini/agent_framework_gemini/_chat_client.py, python/packages/gemini/pyproject.toml
• Tests run: python/packages/gemini/tests/test_issue_6954_repro.py
• Reported version: 1.0.0a260630
• Current version: 1.0.0a260630