fix: skip JSON fallback when web_search tool is present

[bhaskargurram-ai]

Description

Fixes #8958

When using tools with type: "web_search", the ChatAdapter was falling back to JSON mode after the initial structured output attempt failed. This caused a misleading error message that masked the actual underlying error.

Problem

The current error flow was:

1. Attempt structured output → fails with original error
2. Catch exception and fall back to JSON mode
3. JSON mode fails with: "Web Search cannot be used with JSON mode"

This masked the real error from step 1, making debugging very difficult.

Solution

Added detection for web_search tools in lm_kwargs before attempting JSON fallback. When web_search is present, the adapter now skips the JSON fallback and raises the original error instead.

Changes

Modified: dspy/adapters/chat_adapter.py

• Added check for web_search tools before JSON fallback (lines 47-53)
• Detects tools with type: "web_search" in lm_kwargs
• Skips JSON fallback when web_search is detected
• Raises original error for better debugging visibility

# Check if web_search tool is present - if so, don't fall back to JSON mode
# because OpenAI doesn't support JSON mode with web search
tools = lm_kwargs.get('tools', [])
has_web_search = any(tool.get('type') == 'web_search' for tool in tools if isinstance(tool, dict))

if has_web_search:
    # Don't fall back to JSON mode with web search - raise the original error
    raise e

Why This Improves Error Handling

Before this fix:

WARNING: Failed to use structured output format, falling back to JSON mode.
ERROR: Web Search cannot be used with JSON mode

❌ Confusing - makes developers think the issue is JSON mode compatibility

After this fix:

WARNING: Failed to use structured output format, falling back to JSON mode.
ERROR: AttributeError: 'dict' object has no attribute 'type'

✅ Clear - shows the actual error that needs to be addressed

Testing

Tested with a reproduction script:

• Created LM with tools=[{"type": "web_search"}]
• Attempted to generate response
• Verified JSON fallback is skipped when web_search is present
• Confirmed original error is now visible

Related Discussion

This addresses the concern raised by @chenmoneygithub:

"Maybe we should include a configurable argument in ChatAdapter to skip JSON retry? Default to JSON retry is useful, but sometimes this hits me when debugging DSPy programs because JSON retry hides the error stack"

This implementation automatically detects web_search and skips the retry, solving the issue without requiring manual configuration.

Additional Notes

Users may still encounter errors when using web_search (e.g., if using incompatible model types), but they'll now see the actual error instead of a misleading JSON mode error. This significantly improves the debugging experience.

[TomeHirata]

I don't think the issue is limited to the specific tool. Can we just add a warning log in the fallback block?

[bhaskargurram-ai]

@TomeHirata You're absolutely right - a warning log is a much better approach!

I've updated the PR to add a warning log that displays the original error before attempting JSON fallback. This:

• ✅ Preserves the existing fallback behavior (no breaking changes)
• ✅ Helps with debugging regardless of which tool is causing issues
• ✅ Shows both the original error and the fallback error when both fail

The warning now shows:

WARNING: Structured output failed with error: AttributeError: 'dict' object has no attribute 'type'. Falling back to JSON mode.

This gives developers full context to debug the root cause. Thanks for the feedback!

[TomeHirata]

Why are there so many diffs? I only expect one line addition for a warning.

[TomeHirata]

Let's defind the logger on the module level

[TomeHirata]

Let's call it as JSONAdapter to clarify

[TomeHirata]

We can simply use e without custom formatting