bind_tools NotImplementedError when using ChatOllama #21479

hyhzl · 2024-05-09T11:30:30Z

Checked other resources

I added a very descriptive title to this issue.
I searched the LangChain documentation with the integrated search.
I used the GitHub search to find a similar question and didn't find it.
I am sure that this is a bug in LangChain rather than my code.
The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

def init_ollama(model_name:str = global_model):
# llm = Ollama(model=model_name)
llm = ChatOllama(model=model_name)
return llm

llm = init_ollama()
llama2 = init_ollama(model_name=fallbacks)
llm_with_fallbacks = llm.with_fallbacks([llama2])

def agent_search():
search = get_Tavily_Search()
retriver = get_milvus_vector_retriver(get_webLoader_docs("https://docs.smith.langchain.com/overview"),global_model)
retriver_tool = create_retriever_tool(
retriver,
"langsmith_search",
"Search for information about LangSmith. For any questions about LangSmith, you must use this tool!",
)
tools = [search,retriver_tool]
# llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0) # money required
prompt = hub.pull("hwchase17/openai-functions-agent")
agent = create_tool_calling_agent(llm,tools,prompt) # no work.
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({"input": "hi!"})

Error Message and Stack Trace (if applicable)

Traceback (most recent call last):
File "agent.py", line 72, in
agent = create_tool_calling_agent(llm,tools,prompt)
File "/home/anaconda3/envs/languagechain/lib/python3.8/site-packages/langchain/agents/tool_calling_agent/base.py", line 88, in create_tool_calling_agent
llm_with_tools = llm.bind_tools(tools)
File "/home/anaconda3/envs/languagechain/lib/python3.8/site-packages/langchain_core/language_models/chat_models.py", line 912, in bind_tools
raise NotImplementedError()
NotImplementedError

Description

because ollama provide great convenient for developers to develop and practice LLM app, so hoping this issue to be handled as soon as possible
Appreciate sincerely !

System Info

langchain==0.1.19
platform: centos
python version 3.8.19

sbusso · 2024-05-09T12:56:36Z

@hyhzl, no random mention, please.

subhash137 · 2024-05-10T11:44:51Z

Any one gt the solution for that

subhash137 · 2024-05-10T11:51:02Z

Even structuredoutut is not working

error -

tcztzy · 2024-05-11T07:41:46Z

You can use Ollama's OpenAI compatible API like

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    api_key="ollama",
    model="llama3",
    base_url="http://localhost:11434/v1",
)
llm = llm.bind_tools(tools)

Pretend the Ollama mode as OpenAI and have fun with the LLM developping!

subhash137 · 2024-05-11T08:52:46Z

Thank you for your reply

…

On Sat, 11 May 2024, 13:12 Tang Ziya, ***@***.***> wrote: You can use Ollama's OpenAI compatible API <https://ollama.com/blog/openai-compatibility> like from langchain_openai import ChatOpenAIllm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", )llm = llm.bind_tools(tools) Pretend the Ollama mode as OpenAI and have fun with the LLM developping! — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHNN3T6STIR6I6LHJSTZBXDVBAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGYYTQMRTG4> . You are receiving this because you commented.Message ID: ***@***.***>

alexanderp99 · 2024-05-13T13:53:55Z

@subhash137 . According to the Ollama docs, their Chat Completions API does not support function calling yet. Did you have any success?

subhash137 · 2024-05-13T16:59:22Z

Yes, I did.

…

On Mon, 13 May 2024, 19:24 Alexander, ***@***.***> wrote: @subhash137 <https://github.com/subhash137> . According to the Ollama docs <https://github.com/ollama/ollama/blob/main/docs/openai.md>, their Chat Completions API does not support function calling yet. Did you have any success? — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHLMHTGSQSIIM7XMBDDZCDAYTAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBXGY2DCNZXGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

alexanderp99 · 2024-05-14T06:40:36Z

@subhash137 would you please show, how you achieved function calling in that way?

kaminwong · 2024-05-14T07:46:42Z

@subhash137 would you please show, how you achieved function calling in that way?

tcztzy's comment should work

subhash137 · 2024-05-14T07:53:43Z

from langchain_openai import ChatOpenAI llm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", ) llm = llm.bind_tools(tools) If you want to run locally use LM studio download the models , run the server give the api to base_url. but i prefer to use groq for faster and efficient output.

…

On Tue, 14 May 2024 at 13:17, KM ***@***.***> wrote: @subhash137 <https://github.com/subhash137> would you please show, how you achieved function calling in that way? tcztzy's comment should work — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHNVDN2RXAOE2HL4Z3LZCG6PTAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZGUYDMNBQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

kaminwong · 2024-05-14T10:51:01Z

@subhash137 would you please show, how you achieved function calling in that way?

Oh sorry I just tried, seems the tools are not invoked this way. Did someone successfully make the model use the tools provided?

subhash137 · 2024-05-14T10:56:55Z

Oh 😳 , I am sorry I didn't implemented it correctly. Just now I ran the code it is running successfully but tools are not invoked. Aa.. I don't have any idea now

…

On Tue, 14 May 2024, 16:21 KM, ***@***.***> wrote: @subhash137 <https://github.com/subhash137> would you please show, how you achieved function calling in that way? Oh sorry I just tried, seems the tools are not invoked this way. Did someone successfully make the model use the tools provided? — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHNRKDUWHP6VCW45263ZCHUCZAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZHA4DQMRRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

AmirMohamadBabaee · 2024-05-16T15:49:49Z

I faced this error too. is there any quick fix for this problem? using OllamaFunctions can fix it?

According to this github issue: langchain-ai/langchain#21479 this setup: from langchain_openai import ChatOpenAI class Person(BaseModel): name: str = Field(description="The person's name", required=True) height: float = Field(description="The person's height", required=True) hair_color: str = Field(description="The person's hair color") prompt = PromptTemplate.from_template( """Alex is 5 feet tall. Claudia is 1 feet taller than Alex and jumps higher than him. Claudia is a brunette and Alex is blonde. Human: {question} AI: """ ) llm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", ) llm = llm.with_structured_output(Person) chain = prompt | structured_llm chain.invoke("Describe Alex") Should work, but unfortunately it does not which is why it will be removed from this example.

lalanikarim · 2024-05-27T14:36:55Z

#20881 (merged) already added bind_tools feature into OllamaFunctions
#21625 (pending merge) adds support for tool_calls

Harsh-Kesharwani · 2024-05-28T07:46:17Z

@lalanikarim can i use chat model along with function calling. As i see chatOllama does not supports bind_tools, but in documentation it is given how to bind_tools with chatOllama.

ErfanMomeniii · 2024-05-28T10:33:43Z

We should use OllamaFunctions and pass the LLaMA model name as a parameter, as it includes a suitable bind_tools method for adding tools to the chain. The ChatOllama class does not possess any methods for this purpose.
for more details see https://python.langchain.com/v0.1/docs/integrations/chat/ollama_functions
Alternatively, we can manage this manually by defining new classes that inherit from ChatOllama, incorporating tools as parameters, and creating an appropriate invoke function to utilize these tools.
@Harsh-Kesharwani

lalanikarim · 2024-05-28T14:11:52Z

@lalanikarim can i use chat model along with function calling. As i see chatOllama does not supports bind_tools, but in documentation it is given how to bind_tools with chatOllama.

@Harsh-Kesharwani
Like @ErfanMomeniii suggested, you can use OllamaFunctions if you need function calling capabilities with Ollama.
OllamaFunctions inherits from ChatOllama and adds newer bind_tools and with_structured_output functions as well as adds tool_calls property to AIMessage.
While you can currently already use OllamaFunctions for function calling, there is an unmerged PR #21625 that fixes the issue where you want a chat response from OllamaFunctions in case none of the provided functions are appropriate for the request. I am hoping that to be merged sometime this week.

ntelo007 · 2024-06-04T20:45:47Z

Can someone please post a mini example of tool calling with these pr merges?

KIC · 2024-06-09T13:15:47Z

I am still not able to get it to work:

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool


@tool
def magic_function(input: int):
    """applies magic function to an input"""
    return input * -2

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad")
    ]
)

tools = [magic_function]

model = OllamaFunctions(
    model="llama3",
    # formal="json",    # commented or not, does not change the error
    keep_alive=-1,
    temperature=0,
    max_new_tokes=512,
)

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input":"What is the value of magic_function(3)"})

TypeError: Object of type StructuredTool is not JSON serializable

lalanikarim · 2024-06-09T14:30:34Z

I am still not able to get it to work:

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool


@tool
def magic_function(input: int):
    """applies magic function to an input"""
    return input * -2

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad")
    ]
)

tools = [magic_function]

model = OllamaFunctions(
    model="llama3",
    # formal="json",    # commented or not, does not change the error
    keep_alive=-1,
    temperature=0,
    max_new_tokes=512,
)

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input":"What is the value of magic_function(3)"})

TypeError: Object of type StructuredTool is not JSON serializable

This PR fixed the JSON serialization error and couple other things.

#22339

lalanikarim · 2024-06-10T06:48:37Z

Example notebook with tool calling from withing LangGraph agent.
https://github.com/lalanikarim/notebooks/blob/main/LangGraph-MessageGraph-OllamaFunctions.ipynb

Since #22339 is not yet merged, the notebook installs langchain-expermental from my repo (source for #22339).

Harsh-Kesharwani · 2024-06-10T09:39:50Z

@lalanikarim does agent carries context that is return by tool at every iteration, suppose i have 3 tools below is the execution flow:

agent...
use tool 1
tool 1 response: resp1

agent... (does agent carries resp1 or summarization or knowledge graph for it)
use tool 2
tool 2 response: resp2

agent... (does agent carries resp1, resp2 or summarization or knowledge graph for it)
use tool 3
tool 3 response: resp3

The question is does agent carries tool response as a context for next iteration.

lalanikarim · 2024-06-10T12:00:41Z

@lalanikarim does agent carries context that is return by tool at every iteration, suppose i have 3 tools below is the execution flow:

agent...
use tool 1
tool 1 response: resp1

agent... (does agent carries resp1 or summarization or knowledge graph for it)
use tool 2
tool 2 response: resp2

agent... (does agent carries resp1, resp2 or summarization or knowledge graph for it)
use tool 3
tool 3 response: resp3

The question is does agent carries tool response as a context for next iteration.

@Harsh-Kesharwani

You provide an initial state on every irritation. Unless you pass the previous context into the next iteration, the agent starts with a fresh state every time. I hope this answers your question.

initial_state = ...
updated_state = agent.invoke(initial_state)

next_initial_state = <combine updated_state and a new initial state>
updated_state = agent.invoke(next_initial_state)

Harsh-Kesharwani · 2024-06-10T13:40:17Z

@lalanikarim can i log the prompt which is passed to the agent.

lalanikarim · 2024-06-10T14:14:09Z

@lalanikarim can i log the prompt which is passed to the agent.

@Harsh-Kesharwani
I have included langtrace links for multiple runs in the notebook. Take a look and let me know if that answers your questions.

dosubot bot added Ɑ: core Related to langchain-core 🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature labels May 9, 2024

baswenneker mentioned this issue May 13, 2024

[experimental][llms][OllamaFunctions] Add bind_tools and with_structured_output functions to OllamaFunctions #20881

Merged

hinthornw added 04 new feature New functionality (use for larger scope enhancements) and removed 🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature labels May 15, 2024

AnaCoda mentioned this issue May 26, 2024

[experimental][llms][ollama_functions] Update OllamaFunctions to send tool_calls attribute #21625

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bind_tools NotImplementedError when using ChatOllama #21479

bind_tools NotImplementedError when using ChatOllama #21479

hyhzl commented May 9, 2024 •

edited

sbusso commented May 9, 2024

subhash137 commented May 10, 2024

subhash137 commented May 10, 2024

tcztzy commented May 11, 2024

subhash137 commented May 11, 2024 via email

alexanderp99 commented May 13, 2024

subhash137 commented May 13, 2024 via email

alexanderp99 commented May 14, 2024

kaminwong commented May 14, 2024

subhash137 commented May 14, 2024 via email

kaminwong commented May 14, 2024

subhash137 commented May 14, 2024 via email

AmirMohamadBabaee commented May 16, 2024

lalanikarim commented May 27, 2024 •

edited

Harsh-Kesharwani commented May 28, 2024

ErfanMomeniii commented May 28, 2024 •

edited

lalanikarim commented May 28, 2024

ntelo007 commented Jun 4, 2024

KIC commented Jun 9, 2024 •

edited

lalanikarim commented Jun 9, 2024

lalanikarim commented Jun 10, 2024

Harsh-Kesharwani commented Jun 10, 2024

lalanikarim commented Jun 10, 2024 •

edited

Harsh-Kesharwani commented Jun 10, 2024

lalanikarim commented Jun 10, 2024

bind_tools NotImplementedError when using ChatOllama #21479

bind_tools NotImplementedError when using ChatOllama #21479

Comments

hyhzl commented May 9, 2024 • edited

Checked other resources

Example Code

Error Message and Stack Trace (if applicable)

Description

System Info

sbusso commented May 9, 2024

subhash137 commented May 10, 2024

subhash137 commented May 10, 2024

tcztzy commented May 11, 2024

subhash137 commented May 11, 2024 via email

alexanderp99 commented May 13, 2024

subhash137 commented May 13, 2024 via email

alexanderp99 commented May 14, 2024

kaminwong commented May 14, 2024

subhash137 commented May 14, 2024 via email

kaminwong commented May 14, 2024

subhash137 commented May 14, 2024 via email

AmirMohamadBabaee commented May 16, 2024

lalanikarim commented May 27, 2024 • edited

Harsh-Kesharwani commented May 28, 2024

ErfanMomeniii commented May 28, 2024 • edited

lalanikarim commented May 28, 2024

ntelo007 commented Jun 4, 2024

KIC commented Jun 9, 2024 • edited

lalanikarim commented Jun 9, 2024

lalanikarim commented Jun 10, 2024

Harsh-Kesharwani commented Jun 10, 2024

lalanikarim commented Jun 10, 2024 • edited

Harsh-Kesharwani commented Jun 10, 2024

lalanikarim commented Jun 10, 2024

hyhzl commented May 9, 2024 •

edited

lalanikarim commented May 27, 2024 •

edited

ErfanMomeniii commented May 28, 2024 •

edited

KIC commented Jun 9, 2024 •

edited

lalanikarim commented Jun 10, 2024 •

edited