Local Language Models from Ollama not recognizing tools in crewai

[nishkaloc]

I get the below error
Crewai when using ollama isn't recognizing how to use the tools defined.

> Entering new CrewAgentExecutor chain...
Thought: I now can give a great answer
Action: Internet Search Tool(user_query) - Scrapes the weblinks for the query based on user input.
Action Input: {"query": "who is Virat Kohlis son?"}

Action 'Internet Search Tool(user_query) - Scrapes the weblinks for the query based on user input.' don't exist, these are the only available Actions:
 Internet Search Tool: Internet Search Tool(user_query) - Scrapes the weblinks for the query based on user input.
Returns:
List[str]: A list containing information about the query.

It seems when I am giving the local llm from ollama I face this issue, once I replace it with openai everything works perfectly.
Any suggestion on how to fix it?
I define the local llm as follows

from langchain_community.llms import Ollama

os.environ["OPENAI_API_KEY"]="NA"

llm = ChatOpenAI(
        model="phi3",
        base_url="http://localhost:11434/v1"
    )

I am trying this and been stuck at it since long time.
Please help me fix it.

[JTMarsh556]

I am having the same issue. I dropped my projects weeks ago hoping this would have been fixed as I have seen a lot of claims on youtube that this runs locally but I am beginning to think either something is broken or there is a knowledge gap that is not covered in the documentation.

I followed the crewai-llama2 example thinking that could be the issue but it still doesnt work. Same code works just fine when I flip it back over to OpenAI but it is so token exhausting it doesn't make sense to use openAI.

[MrSimonC]

If it helps (I'm only in my first week of trying this locally, so I'm super new), but I have my first example - similar to yours - working. A few things: Ollama is OpenAI compatible (so no need for langchain tool import) [apart from embeddings - so we can't use memory=True in the Crew() just yet]. I also used a free Bing API key for mine - just to try something new.

#!python3
import os
import warnings
from crewai import Agent, Task, Crew
from crewai_tools import ScrapeWebsiteTool
from langchain.agents import Tool
from langchain.utilities.bing_search import BingSearchAPIWrapper
from pydantic import BaseModel

warnings.filterwarnings('ignore')

os.environ["OPENAI_API_BASE"]='http://localhost:11434/v1'
os.environ["OPENAI_MODEL_NAME"]='phi3'  # Adjust based on available model
os.environ["OPENAI_API_KEY"] = "NA"
os.environ["BING_SUBSCRIPTION_KEY"] = "<PUT YOUR (FREE) BING SUBSCRIPTION KEY HERE>"
os.environ["BING_SEARCH_URL"] = "https://api.bing.microsoft.com/v7.0/search"

search = BingSearchAPIWrapper()
scrape_tool = ScrapeWebsiteTool()

bing_search = Tool(
  name="Search",
  func=search.run,
  description="Useful for search-based queries"
)

researcher = Agent(
  role='Senior Research Analyst',
  goal='Using the Search tool, provide up to date data and analysis on {topic}',
  backstory='An expert researcher who can convey accurate facts about {topic}.',
  tools=[bing_search, scrape_tool]
)

class EventDetails(BaseModel):
    event_date: str
    event_name: str
    summary: str

find_trends = Task(
    description=("Find the dates and summary descriptions of {topic}"),
    expected_output="A date of the upcoming event, a "
        "sentence of the event name, a small summary of {topic}.",
    output_json=EventDetails,
    output_file="event_details.json",
    agent=researcher
)

crew = Crew(
    agents=[researcher],
    tasks=[find_trends],
    verbose=2
)

result = crew.kickoff(inputs={"topic": "The next Apple event in 2024"})

[JTMarsh556]

I have tried a few variations of that based on what I found online but they all break when I have more than one agent. There is no communication between them. I don't recall the specific error but it essentially says that the input is wrong between agents. Either way I will give this variation a try. I sincerely appreciate you taking the time to post this response.

[jcoombes]

@MrSimonC fyi this pull request exists if you want to use Memory=True with ollama, and then return to the main branch when this is integrated.

#439 (comment)

[alphaveneno]

https://www.youtube.com/watch?v=0ai-L50VCYU&t=16s

codewithbrandon has a tutorial on adapting open-source LLMs to crewai, it includes the code

You need to prepare the LLMs for crewai. This example is for adapting llama3
There are two files needed:

---

create-llama3-model-file.sh

#!/bin/bash

# Variables
model_name="llama3"
custom_model_name="crewai-llama3"

# Get the base model
ollama pull $model_name

# Create the model file
ollama create $custom_model_name -f ./Llama3Modelfile

---

Llama3Modelfile.txt

FROM llama3

# set parameters

PARAMETER temperature 0.8
PARAMETER stop Result

# Sets a custom system message to specify the behavior of 
# the chat assistant

# Leaving it blank for now.

SYSTEM """"""

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.