[Bug]: Anthropic Extended Thinking With Tool Use Returns, `Thinking may not be enabled when tool_choice forces tool use`

[Kaushal-26]

Bug Description

• Using Anthropic extended thinking to return structured output gives this error: Thinking may not be enabled when tool_choice forces tool use
• This also comes up when we force tool_required=True for completions.
• As per this documentation: https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#extended-thinking-with-tool-use

  Tool choice limitation: Tool use with thinking only supports tool_choice: {"type": "auto"} (the default) or tool_choice: {"type": "none"}. Using tool_choice: {"type": "any"} or tool_choice: {"type": "tool", "name": "..."} will result in an error because these options force tool use, which is incompatible with extended thinking.

• While, llama index is sending tool_choice: {"type": "any"}

Version

0.13.0

Steps to Reproduce

Example Code from https://docs.llamaindex.ai/en/stable/examples/llm/anthropic/#structured-prediction and https://docs.llamaindex.ai/en/stable/examples/llm/anthropic/#model-thinking:

from typing import List

from llama_index.llms.anthropic import Anthropic
from llama_index.core.prompts import PromptTemplate
from llama_index.core.bridge.pydantic import BaseModel

from dotenv import load_dotenv
load_dotenv()

class MenuItem(BaseModel):
    """A menu item in a restaurant."""

    course_name: str
    is_vegetarian: bool


class Restaurant(BaseModel):
    """A restaurant with name, city, and cuisine."""

    name: str
    city: str
    cuisine: str
    menu_items: List[MenuItem]


llm = Anthropic(
    model="claude-sonnet-4-0",
    # max_tokens must be greater than budget_tokens
    max_tokens=64000,
    # temperature must be 1.0 for thinking to work
    temperature=1.0,
    thinking_dict={"type": "enabled", "budget_tokens": 1600},
)

prompt_tmpl = PromptTemplate(
    "Generate a restaurant in a given city {city_name}"
)

restaurant_obj = (
    llm.as_structured_llm(Restaurant)
    .complete(prompt_tmpl.format(city_name="Miami"))
    .raw
)

print(restaurant_obj)

Relevant Logs/Tracbacks

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/Projects/amall/main.py", line 42, in <module>
    .complete(prompt_tmpl.format(city_name="Miami"))
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 317, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py", line 435, in wrapped_llm_predict
    f_return_val = f(_self, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/structured_llm.py", line 95, in complete
    return complete_fn(prompt, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/base/llms/generic_utils.py", line 184, in wrapper
    chat_response = func(messages, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 317, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py", line 175, in wrapped_llm_chat
    f_return_val = f(_self, messages, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/structured_llm.py", line 63, in chat
    output = self.llm.structured_predict(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 317, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/llm.py", line 360, in structured_predict
    result = program(llm_kwargs=llm_kwargs, **prompt_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 317, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/program/function_program.py", line 151, in __call__
    agent_response = self._llm.predict_and_call(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 317, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/function_calling.py", line 228, in predict_and_call
    response = self.chat_with_tools(
               ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/function_calling.py", line 55, in chat_with_tools
    response = self.chat(**chat_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index_instrumentation/dispatcher.py", line 317, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py", line 175, in wrapped_llm_chat
    f_return_val = f(_self, messages, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/llama_index/llms/anthropic/base.py", line 393, in chat
    response = self._client.messages.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/anthropic/_utils/_utils.py", line 283, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/anthropic/resources/messages/messages.py", line 997, in create
    return self._post(
           ^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/anthropic/_base_client.py", line 1324, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/Projects/amall/.venv/lib/python3.11/site-packages/anthropic/_base_client.py", line 1112, in request
    raise self._make_status_error_from_response(err.response) from None
anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Thinking may not be enabled when tool_choice forces tool use.'}}

[logan-markewich]

:sad: that sucks. This means structured outputs will be less reliable with thinking, fyi (since we cant guarantee the tool call)

The fix would be detecting if thinking is enabled before setting the tool choice automatically

[Kaushal-26]

:sad: that sucks. This means structured outputs will be less reliable with thinking, fyi (since we cant guarantee the tool call)

The fix would be detecting if thinking is enabled before setting the tool choice automatically

Yeah, you are correct. We can't guarantee the tool call, but as thinking is involved, this will not happen. I found this in another repo, which is also occurring in the llama index.

Will add a fix soon