Are there some models in ollama can support function calling or object return?

[chaos369]

No description provided.

[jackmpcollins]

@chaos369 Looks like ollama does support tool calls, but litellm currently has some bugs with how it is parsing these. I've opened an issue there BerriAI/litellm#3333 When that is fixed you should be able to use ollama models via the LitellmChatModel in magentic by using the "ollama_chat/" prefix (instead of "ollama/") in the model name.

Something like this (based on example from #50 (comment))

from magentic import prompt
from magentic.chat_model.litellm_chat_model import LitellmChatModel


@prompt(
    "Count to 5",
    model=LitellmChatModel("ollama_chat/llama2", api_base="http://localhost:11434")
)
def test() -> list[int]: ...


test()

@knoopx you might be interested in following this issue

[chaos369]

Thank you.

[jackmpcollins]

I've opened a PR on litellm that should fix this BerriAI/litellm#3469 Just need to bump the litellm dependency version in magentic once that's merged and released.

[jackmpcollins]

@chaos369 @DevAseel I have just published https://github.com/jackmpcollins/magentic/releases/tag/v0.23.0 which enables structured outputs and function calling with ollama 🦙 Depending on what model you use you might need to prompt it to "use the tool" or add more details of the response format in the prompt.

Example

from magentic import prompt
from magentic.chat_model.litellm_chat_model import LitellmChatModel


@prompt(
    "Count to {n}. Use the tool to return in the format [1, 2, 3, ...]",
    model=LitellmChatModel("ollama_chat/llama2", api_base="http://localhost:11434")
)
def count_to(n: int) -> list[int]: ...


count_to(5)
# > [1, 2, 3, 4, 5]

Please let me know if you hit any issues with this. Thanks

[chaos369]

It seems hardly to return an object. @jackmpcollins

code for testing

wizardlm2=LitellmChatModel("ollama/wizardlm2:7b")
llama3 = LitellmChatModel("ollama/llama3:instruct")
gpt35=LitellmChatModel(model="openai/gpt-3.5-turbo")

llm_model = wizardlm2

class Superhero(BaseModel):
name: str
age: int
power: str
enemies: list[str]

@prompt("Create a Superhero named {name}, use the tool to return a Superhero struct.", model=llm_model)
def create_superhero(name: str) -> Superhero: ...

hero = create_superhero("Garden Man")

the result when "llm_model = wizardlm2"

Traceback (most recent call last):
File "/home/mac/workspace/projects/mag_hero.py", line 23, in
hero = create_superhero("Garden Man")
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/prompt_function.py", line 81, in call
message = self.model.complete(
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/litellm_chat_model.py", line 176, in complete
content = next(parse_streamed_tool_calls(response, tool_schemas))
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/openai_chat_model.py", line 266, in parse_streamed_tool_calls
tool_schema = select_tool_schema(first_chunk, tool_schemas)
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/openai_chat_model.py", line 215, in select_tool_schema
raise ValueError(msg)
ValueError: Unknown tool call: {"id":"call_6618f929-fcd0-4a54-9cfe-48c3dcedf42f","function":{"arguments":"{"argument_name": "Garden Man", "power": "Ability to control and manipulate plants, flora, and fauna; can also create powerful plant-based weapons.", "enemies": ["Pollutants", "Urban Developers", "Invasive Species Villains"], "age": 35}","name":"createGardenMan"},"type":"function","index":0}

the result when "llm_model = llama3"

Traceback (most recent call last):
File "/home/mac/workspace/projects/mag_hero.py", line 23, in
hero = create_superhero("Garden Man")
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/prompt_function.py", line 81, in call
message = self.model.complete(
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/litellm_chat_model.py", line 154, in complete
first_chunk = next(response)
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/litellm/llms/ollama.py", line 297, in ollama_completion_stream
raise e
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/litellm/llms/ollama.py", line 256, in ollama_completion_stream
status_code=response.status_code, message=response.text
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/httpx/_models.py", line 574, in text
content = self.content
File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/httpx/_models.py", line 568, in content
raise ResponseNotRead()
httpx.ResponseNotRead: Attempted to access streaming response content, without having called read().

[jackmpcollins]

@chaos369 I think the ResponseNotRead issue here could be caused by Ollama. Could you make sure it is fully up-to-date and that you have the llama3 model pulled/downloaded.

The first issue is caused by the model returning an invalid function name, createGardenMan instead of return_superhero which is expected by magentic. I've tried several prompts with llama3 to fix this but it is very unreliable. I think the solution will required updates to ollama and litellm. I've opened a new magentic issue to track this #207 so please follow that.