Tutorial - Running Autogen using a local LLM running in oobabooga WebUI served via LiteLLM

[deronkel82]

Suggesting you have Python, Autogen and oobabooga WebUI installed and running fine:

1. Install LiteLLM

pip install litellm 

2. Install the openai API extension in the oobabooga WebUI

In the folder where the WebUI is installed:

cd openai
pip install -r requirements.txt 

Run the WebUI with the openai extension enabled:

python server.py --extensions openai 

3. Create a python file (f.e. test.py) with the following code:

import openai

openai.api_key ='sk-1111111111111111111111111111111' # no real api-key needed!

response = completion(
  model="oobabooga/TheBloke_WizardCoder-Python-13B-V1.0-GPTQ",
  messages=[{ "content": "Hi! How are you?","role": "user"}],
  api_base="http://127.0.0.1:5001/api",
  custom_llm_provider="openai",
  max_tokens=4000
)

print(response['choices'][0]['message']['content']) 

4. Run LiteLLM to serve the LLM-Model loaded in the WebUI:

litellm --model oobabooga/TheBloke_WizardCoder-Python-13B-V1.0-GPTQ

In this example the model TheBloke_WizardCoder-Python-13B-V1.0-GPTQ is used.

5. Now run the python file

python test.py

You should get a useful response to the question "Hi! How are you?".

6. If it works as intended the fun part begins. Create a new pyhton file (f.e. test-autogen.py) with the following code:

from autogen import AssistantAgent, UserProxyAgent, oai, GroupChat, GroupChatManager
config_list=[
    {
        "model": "oobabooga/TheBloke_WizardCoder-Python-13B-V1.0-GPTQ",
        "api_base": "http://127.0.0.1:5001/api",
        "api_type": "open_ai",
        "api_key": "NULL", # required, you don´t need to change this
        }
]


llm_config={
    "config_list": config_list,
}

coder = AssistantAgent(
    name="Coder",
    system_message="You write the code to solve the required Task. Implement improvements provided by the critic.",
    llm_config=llm_config,
)

critic = AssistantAgent(
    name="critic",
    system_message="You will make sure that the code provided by coder is bug free and all requirements are met. also provide coder with ideas for more features that make the project better. do not provide any code.",
    llm_config=llm_config
)

user_proxy = UserProxyAgent(
    name="user_proxy",
    system_message="A human admin wo will give the idea and run the code provided by Coder.",
    human_input_mode="ALWAYS",
    is_termination_msg=lambda x: x.get("content", "").rstrip().endswith("TERMINATE"),
)

groupchat = GroupChat(agents=[user_proxy, coder, critic], messages=[], max_round=12)
manager = GroupChatManager(groupchat=groupchat, llm_config=llm_config)
user_proxy.initiate_chat(manager, message="Write a calculator app using tkinter for the gui. the GUI Layout should resemble the layout of a casio calculator.", clear_history=False, config_list=config_list)

If everything works fine a groupchat should begin with the task to write code for a calculator app.

7. Possible problems:

I had multiple errors where the script ran into timeouts. If you have the same problem here is the fix that worked for me. I used the Pycharm IDE so the filepaths maybe different for you but the filenames are the same:

Locate the file api_requestor.py (path on my system: venv/Lib/site-packages/openai/api_requestor.py) and change the "TIMEOUT_SECS = 600" to 6000.

In the file completition.py (path on my system: venv/Lib/site-packages/autogen/oai/completion.py) change the "request_timeout = 60" to 600.

8. Have fun!!

[sonichi]

Thanks. request_timeout can be set without modifying autogen code. https://microsoft.github.io/autogen/docs/FAQ#handle-rate-limit-error-and-timeout-error