Using Model from Ollama in ChatOpenAI doesnt invoke the tools with bind_tools

[AtmehEsraa]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    api_key="ollama",
    model="llama3:8b-instruct-fp16",
    base_url="http://localhost:11434/v1",
)

Description

Using Model from Ollama in ChatOpenAI doesnt invoke the tools with bind_tools

System Info

..

[dosubot[bot]]

To address the issue of invoking tools with bind_tools when using the Ollama model in ChatOpenAI, ensure you're correctly binding your tools to the chat model. Here's a concise guide:

1. Bind Tools Correctly: Use the bind_tools method to attach your tools to the ChatOpenAI instance. This involves creating tool instances and converting them into a format that ChatOpenAI can use. For example:

from langchain_openai import ChatOpenAI
from langchain.tools import MoveFileTool, format_tool_to_openai_function

llm = ChatOpenAI(
    api_key="ollama",
    model="llama3:8b-instruct-fp16",
    base_url="http://localhost:11434/v1",
)

tools = [MoveFileTool()]
functions = [format_tool_to_openai_function(t) for t in tools]

llm_with_tools = llm.bind_tools(tools=functions)

2. Check Tool Compatibility: Ensure your tools are compatible with ChatOpenAI. Tools can be dictionaries, Pydantic models, callable functions, or BaseTool instances.

3. Error Handling: If you encounter errors, verify that you're not passing unsupported arguments to bind_tools and that your tools are correctly defined and formatted.

4. Structured Output and Schema: When using structured output, ensure a valid schema is specified. This can be a Pydantic class or a dictionary.

If following these steps doesn't resolve your issue, please provide more details about the errors or behavior you're experiencing for further assistance.

Details

Sources

• AttributeError: 'ChatGoogleGenerativeAI' object has no attribute 'bind_tools'
• Functions not called when i use langchain_openai.ChatOpenAI as llm
• Tools within ChatModel
• Issue with Custom Tools Invocation and Auto-generated Responses from Langchain LLM
• ChatOpenAI with "bind_tools", If use "base_url" other API sever, dont call "tool" and don't response "tool_calls"
• libs/partners/openai/langchain_openai/chat_models/base.py
• libs/experimental/langchain_experimental/llms/ollama_functions.py
• libs/partners/openai/tests/integration_tests/chat_models/test_base.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[maximeperrindev]

Hi @AtmehEsraa,

You cannot use ChatOpenAI with an Ollama model.
Moreover, Ollama chat model does not support tool calling : https://python.langchain.com/v0.1/docs/integrations/chat/

[AtmehEsraa]

@maximeperrindev so how we can solve this issue ?

[maximeperrindev]

@AtmehEsraa,

Well, if you want to use Ollama you can't. The workaround will be to use one of the models supporting tool calling.

[romilandc]

@maximeperrindev That's incorrect. You CAN use ChatOpenAI with Ollama you must pull model via Ollama first however. OP's code from langchain_openai import ChatOpenAI llm = ChatOpenAI( api_key="ollama", model="llama3:8b-instruct-fp16", base_url="http://localhost:11434/v1", ) should work as long as he has the an Ollama instance active when using ChatOpenAI().

[eyurtsev]

See this guide to add ad-hoc too calling capability to models that do not support it natively: https://python.langchain.com/v0.2/docs/how_to/tools_prompting/

Keep in mind that the gap in quality can be very big between a model that's been fine-tuned for tool calling and one that hasn't.

For tool calling models see:

https://python.langchain.com/v0.2/docs/how_to/tool_calling/

And a list of such models here: https://python.langchain.com/v0.2/docs/integrations/chat/

[cboettig]

Thanks for clarifying this @eyurtsev , super helpful. from those docs:

Tool calling is not universal, but many popular LLM providers, including Anthropic, Cohere, Google, Mistral, OpenAI, and others, support variants of a tool calling feature.

Apologies if this is a naiive question, but do any of the models listed there have support for self-hosting the llm in tool-calling mode?

That is, I know that we can download and run the open model weights for models from Cohere and Mistral via ollama and other mechanisms. I also see ollama-langchain explicitly does not support tooling, though that feels a bit apples-to-oranges as ollama obviously isn't itself a model but only an interface to collection of models, some of which are and some of which are not tuned for tools.

The examples with tool chains for Cohere and Mistral show using the corresponding API keys to access the hosted versions of those models. But I cannot tell if that is the only option for using these models in tool form (i.e. that they cannot be self-hosted in this mode, at least not within existing langchain tooling), or merely because that is the most convenient. Thanks!

[eyurtsev]

Hi @cboettig ,

Not a naive question at all. I don't know which features are supported now by ollama. The langchain integration with ollama definitely does not support tools currently.

According to these docs: https://github.com/ollama/ollama/blob/main/docs/openai.md, JSON mode exists, but not function calling, but this is using the openai format.

If you find more, please share!

[romilandc]

Getting similar error when using llm.with_structured_output() within a function chained with llm.bind_tools(tools).

[yaoduo11]

Is there any good solution?