Python: [Bug]: header_provider not applied to MCP initialize() request

[efranken]

Description

header_provider on MCPStreamableHTTPTool only gets called from call_tool(). The initialize handshake sent during connect() goes out before any tool call happens, so it never gets headers — for MCP servers that require auth on initialize (Azure AI Search knowledge-base MCP endpoints, for example), this means an immediate 401 and the tool never connects.

Root cause is in _mcp.py — _header_provider is only invoked in MCPStreamableHTTPTool.call_tool(), and session.initialize() runs earlier in connect()/_connect_on_owner(), before any tool call could happen.

I haven't contributed to a large repo like this before, attempting to be as good of a steward as possible but may have missed guidance along the way. Happy to make any corrections.

Code Sample

\`\`\`python
def auth_headers(_kwargs: dict) -> dict[str, str]:
    print("called")  # never prints
    return {"Authorization": f"Bearer {get_token()}"}

tool = MCPStreamableHTTPTool(
    name="kb",
    url="https://<search-service>.search.windows.net/knowledgebases/<kb>/mcp?api-version=2025-11-01-Preview",
    header_provider=auth_headers,
)

agent = Agent(client=client, tools=[tool])
await agent.run("hello")  # 401 on initialize, header_provider never runs
\`\`\`

Error Messages / Stack Traces

`Error MCP server failed to initialize: Cancelled via cancel scope #####`

As seen when using a hosted Foundry agent to connect to an Azure AI Search Knowledgebase

Package Versions

agent-framework-core 1.1.0

Python Version

3.1.2

Additional Context

No response

[github-actions]

🤖 Automated triage reproduction notes (agent-authored — trust but verify)

Agent analysis

Repro: MCPStreamableHTTPTool in packages/core/agent_framework/_mcp.py::call_tool (line 3104) is the only location that invokes _header_provider and sets the _mcp_call_headers ContextVar; _connect_on_owner (line 1263) calls session.initialize() (line 1343) before any call_tool, so the httpx event hook _inject_headers (line 3071) reads an empty ContextVar and injects no headers. Minimal repro: construct MCPStreamableHTTPTool with a header_provider, call get_mcp_client(), then fire an HTTP request through tool._httpx_client — the Authorization header is absent.

• Failing test: python/packages/core/tests/core/test_mcp_header_provider_initialize_bug.py
• Files examined: python/packages/core/agent_framework/_mcp.py, python/packages/core/tests/core/test_mcp.py, python/packages/core/pyproject.toml
• Tests run: tests/core/test_mcp_header_provider_initialize_bug.py::test_header_provider_not_called_during_initialize, tests/core/test_mcp_header_provider_initialize_bug.py::test_header_provider_headers_missing_from_initialize_request, tests/core/test_mcp_header_provider_initialize_bug.py::test_header_provider_only_called_in_call_tool_not_connect
• Reported version: 1.1.0
• Current version: 1.11.0

[abhishekabramaina]

Hello everyone,

I did a deep dive into the codebase to trace the root cause and evaluate the safest, most robust fix for this issue. Below is a detailed breakdown of the findings, the design alternatives I evaluated, and the local verification, structured in accordance with the project's contributing guidelines.

1. High-Level Problem & Root Cause

In agent_framework, outgoing HTTP requests for MCPStreamableHTTPTool pass through the _inject_headers request event hook (registered on the httpx.AsyncClient). This hook is responsible for fetching headers from the _mcp_call_headers ContextVar and injecting them into the request.

However, _mcp_call_headers is only set within MCPStreamableHTTPTool.call_tool(). Consequently, any HTTP requests sent outside of a tool call execution context are missing their authentication headers.

This impacts the following lifecycle phases:

• Phase 1: Eager Connection (connect / initialize)
  When the tool context is entered (__aenter__), it triggers connect(), which starts the session.initialize() handshake. Since this runs before any tool calls, the hook injects no headers, leading to a 401 Unauthorized response on authenticated endpoints.

• Phase 2: Lifecycle Updates (load_tools / load_prompts / send_ping)
  Any dynamic updates or background pings executed after connection also run outside call_tool(), so they fail with 401 Unauthorized as well.

• Phase 3: Runtime Execution (call_tool)
  Tool calls succeed because they wrap the execution context and populate the ContextVar.

2. Proposed Design & Alternatives

Recommended Approach: Add an Ambient Fallback in the Subclass Hook

Update the _inject_headers request event hook in MCPStreamableHTTPTool.get_mcp_client() to support an ambient fallback:

headers = _mcp_call_headers.get(None)
if headers is None:
    headers = self._header_provider({})

Benefits

• Encapsulation: Keeps all HTTP-specific header logic contained within MCPStreamableHTTPTool.
• Complete Coverage: Every HTTP request—including handshake, discovery, background pings, and tool calls—automatically receives the appropriate headers.
• Dynamic Refresh: Calls _header_provider({}) on demand for ambient requests, allowing dynamically refreshed credentials (for example, OAuth or managed identity tokens).
• Lazy Connection Support: If the tool connects lazily on the first tool call, the ContextVar is already populated, so the fallback is naturally bypassed.

Alternative 1: Wrap connect_on_owner in the Base Class

Design: Wrap the connection block in _connect_on_owner (defined in _mcp.py) to temporarily populate the ContextVar using getattr(self, "_header_provider").

Drawbacks

• Breaks Encapsulation: _connect_on_owner belongs to the shared MCPTool base class (used by stdio and WebSocket implementations). Accessing _header_provider, which is HTTP-specific, leaks subclass details into the base class.
• Incomplete Coverage: Dynamic refreshes and background pings executed after connect() completes would still run without authentication.

Alternative 2: Configure Default Client Headers During Construction

Design: Call _header_provider({}) during client creation and set the returned headers as the default headers on the httpx.AsyncClient.

Drawbacks

• No Dynamic Token Refresh: The headers become fixed at client creation time, so expiring credentials cannot be refreshed automatically.
• Added Complexity: Call-specific headers would need to override the default client headers, making the event hook more complicated than necessary.

---

Local Verification

I verified the recommended approach locally by:

1. Adding a new unit test, test_mcp_streamable_http_tool_header_provider_injects_on_connect, in python/packages/core/tests/core/test_mcp.py to verify that headers are injected during the connection/handshake phase.
2. Running the full MCP test suite (pytest packages/core/tests/core/test_mcp.py), with all 272 tests passing.
3. Running the formatting and lint checks (uv run poe syntax -P core), which completed successfully.

@efranken and @giles17 @moonbox3 : I'd love to hear your thoughts on this approach.

[efranken]

Appreciate your review. My PR is locally scoped to my issue, whereas your proposal is global and catches more failures than I experienced.

I believe the proposal will encounter a key error when running mcp_api_key_auth.py. it indexes the mcp_api_key which is only present after agent.run, which would happen after connect().

Proposal is to either modify the sample (potentially breaking existing solutions built using the sample) or add a try/except:

                   headers = _mcp_call_headers.get(None)
                   if headers is None:
                       try:
                           headers = self._header_provider({})
                       except Exception:
                           logger.warning(
                               "header_provider raised for MCP server %r on an ambient request; proceeding without headers.",
                               self.name,
                               exc_info=True,
                           )
                           headers = {}

I can modify or happy to defer to another PR.