Skip to content

[Issue]: unable to make calculator example work with local models #2953

Closed
@geoffroy-noel-ddh

Description

@geoffroy-noel-ddh

Describe the issue

No matter which local model I use the calculator example from the autogen Tool Use tutorial fails in various ways:

  1. the agent often fails to produce inputs that match the function signature (e.g. passing extra parameter d or a number instead of the operator, {"a": 44232, "b": 13312, "operator": "/", "c": 232, "d": 32})
  2. even when the agent sometimes matches the function signature the mathematical expression isn't correctly decomposed into the correct elementary binary operations (e.g. {"a": 44232, "b": 13312, "operator": "+"});
  3. the agents are stuck in a loop and never terminate (even if it manages to return the correct result of a simpler expression);
  4. the agents never factor in the result of the calculator function to formulate the next messages (might be the cause for failure 2 above);

I would like to know:

  1. what is the cause of each problem (is it with autogen, local model, the prompt, my configuration, ...?)
  2. if anyone has managed to make the calculator example work with a local model?
  3. if so, how? could a complete working example be provided here or in the tutorial?

More generally the issue for me is that despite the autogen tutorial giving examples and ways to adapt them for local models, those instructions together don't seem to work with a large number of local models. So, perhaps, either the documentation should acknowledge that limitation upfront (and point to the technical reasons) or it should be more specific on how to make the example work. Knowing where the limitations are (autogen, model, litellm or ollama?) would be
very helpful.

Steps to reproduce

  1. take the python code of the calculator example exactly as it is written in the Tool Use tutorial (the first version of the code at the top of the tutorial page)
  2. change llm_config of the assistant to use LiteLLM + Ollama exactly as described in the user guide
  3. configure ollama to use llama3:instruct (or other similar model, e.g. mistral:v0.3, phi3:latest, ...)
  4. run the adapted calculator example

In the output log below which exhibits all the 4 issues described above.

Screenshots and logs

User (to Assistant):

What is (44232 + 13312 / (232 - 32)) * 5?

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Assistant (to User):

***** Suggested tool call (call_a1ff8853-5428-409f-805a-a9e1bf22ca42): calculator *****
Arguments: 
{"a": 44232, "b": 13312, "c": 200, "d": 32, "operator": "*"}
***************************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):

User (to Assistant):

***** Response from calling tool (call_a1ff8853-5428-409f-805a-a9e1bf22ca42) *****
Error: calculator() got an unexpected keyword argument 'c'
**********************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Assistant (to User):

***** Suggested tool call (call_f989593b-36ac-4788-bb22-30162c6462f8): calculator *****
Arguments: 
{"a": 44232, "b": 13312, "operator": "/", "c": "unexpected"}
***************************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):

User (to Assistant):

***** Response from calling tool (call_f989593b-36ac-4788-bb22-30162c6462f8) *****
Error: calculator() got an unexpected keyword argument 'c'
**********************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Assistant (to User):

***** Suggested tool call (call_a602644e-4192-4168-a511-128f1131440e): calculator *****
Arguments: 
{"a": null, "b": null, "operator": null}
***************************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):

User (to Assistant):

***** Response from calling tool (call_a602644e-4192-4168-a511-128f1131440e) *****
Error: Invalid operator
**********************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Assistant (to User):

***** Suggested tool call (call_d839acb3-6898-4ed9-b36f-5c94cf4c46d1): calculator *****
Arguments: 
{"a": 44232, "b": 13312, "operator": "/"}
***************************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):

User (to Assistant):

***** Response from calling tool (call_d839acb3-6898-4ed9-b36f-5c94cf4c46d1) *****
3
**********************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Assistant (to User):

***** Suggested tool call (call_a21cf647-2c00-4702-8288-fcb3fa78ffc7): calculator *****
Arguments: 
{"a": 44232, "b": 13312.0, "operator": "/"}
***************************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):

User (to Assistant):

***** Response from calling tool (call_a21cf647-2c00-4702-8288-fcb3fa78ffc7) *****
3
**********************************************************************************

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
Assistant (to User):

***** Suggested tool call (call_c5121502-c973-44b9-aa2e-f33aebf4f6f5): calculator *****
Arguments: 
{"a": 44232, "b": 13312.0, "operator": "**"}
***************************************************************************************

--------------------------------------------------------------------------------

Additional Information

  • pyautogen==0.2.28
  • OS: Ubuntu 22.04.4 LTS
  • python: 3.10.12
  • litellm: 1.40.0
  • ollama: 0.1.39

Metadata

Metadata

Assignees

No one assigned

    Labels

    0.2Issues which are related to the pre 0.4 codebasemodelsPertains to using alternate, non-GPT, models (e.g., local models, llama, etc.)needs-triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions