Integration with LM studio for running local models

[Zirgite]

Is it possible to integrate with LM studio? As LM studio has the possiblilty to run a local server with open AI API settings.
I know there is support for ollama but as ollama does not have a native Windows support it is out of the question to mess with WSL.
None of the proposed solutions seems to work. I think this should not be closed as there in no working solution that has bewen proposed.

[matsuobasho]

This question has been answered elsewhere. Start LMStudio, choose a model and then Start a Server (in LMStudio).

Then back in CrewAI, you can either proceed like in #69 or
use the following syntax as in #38. Verify the port LMStudio is using (you'll see it in the GUI when you're starting the server):

local_openai_client = OpenAI(
    base_url="http://localhost:1234/v1", api_key="not-needed", temperature=0.4
)

Then use this object for the llm argument for an Agent.

[joaomdmoura]

New docs being created to include detailed explanation on how to integrated with all these different providers

[Zirgite]

This is the code I used to make it work with LM Studio
import os
from crewai import Agent, Task, Crew
from langchain_openai import ChatOpenAI
from langchain_community.tools import DuckDuckGoSearchRun

from dotenv import load_dotenv
from langchain.chat_models.openai import ChatOpenAI

load_dotenv()

Configure the connection to your local LM Studio instance

OPENAI_API_BASE_URL = "http://localhost:1234/v1" # LM Studio local API endpoint
OPENAI_API_KEY = "NA" # May not be necessary for local LM Studio, but check your LM Studio configuration
MODEL_NAME = "NA" # May not be necessary for local LM Studio, but check your LM Studio configuration

Set default values for API URL, key, and model name

default_llm = ChatOpenAI(
openai_api_base=OPENAI_API_BASE_URL,
openai_api_key=OPENAI_API_KEY,
model_name=MODEL_NAME
)

[Zirgite]

import os
from crewai import Agent, Task, Crew
from langchain_openai import ChatOpenAI
from langchain_community.tools import DuckDuckGoSearchRun

from dotenv import load_dotenv
from langchain.chat_models.openai import ChatOpenAI

load_dotenv()

OPENAI_API_BASE_URL = "http://localhost:1234/v1" # LM Studio local API endpoint
OPENAI_API_KEY = "NA" # May not be necessary for local LM Studio, but check your LM Studio configuration
MODEL_NAME = "NA" # May not be necessary for local LM Studio, but check your LM Studio configuration

default_llm = ChatOpenAI(
openai_api_base=OPENAI_API_BASE_URL,
openai_api_key=OPENAI_API_KEY,
model_name=MODEL_NAME
)

example_agent = Agent(
role='Example Agent',
goal='Show how to assign a custom configured LLM',
backstory='You hang out in the docs section of GitHub repos.',
llm=default_llm
)

from langchain.tools import DuckDuckGoSearchRun
search_tool = DuckDuckGoSearchRun()

researcher = Agent(
role='Senior Research Analyst',
goal='Uncover cutting-edge developments in AI and data science',
backstory="""You work at a leading tech think tank.
Your expertise lies in identifying emerging trends.
You have a knack for dissecting complex data and presenting
actionable insights.""",
verbose=True,
allow_delegation=False,
tools=[search_tool],
llm=default_llm

)
writer = Agent(
role='Tech Content Strategist',
goal='Craft compelling content on tech advancements',
backstory="""You are a renowned Content Strategist, known for
your insightful and engaging articles.
You transform complex concepts into compelling narratives.""",
verbose=True,
allow_delegation=True,
llm=default_llm

)

task1 = Task(
description="""Conduct a comprehensive analysis of the latest advancements in AI in 2024.
Identify key trends, breakthrough technologies, and potential industry impacts.
Your final answer MUST be a full analysis report""",
agent=researcher
)

task2 = Task(
description="""Using the insights provided, develop an engaging blog
post that highlights the most significant AI advancements.
Your post should be informative yet accessible, catering to a tech-savvy audience.
Make it sound cool, avoid complex words so it doesn't sound like AI.
Your final answer MUST be the full blog post of at least 4 paragraphs.""",
agent=writer
)

crew = Crew(
agents=[researcher, writer],
tasks=[task1, task2],
verbose=2, # You can set it to 1 or 2 to different logging levels
)

result = crew.kickoff()

print("######################")
print(result)

[Zirgite]

ADDITIONAL EXPLANATIONS
Correct the Connection Settings:
    Update the OPENAI_API_BASE_URL to the correct port (1234 in your case). In the documentation it is a typo mistake  localhost:8000/v1"

Initialize the LLM (Language Learning Model):
    Configure default_llm with the correct LM Studio API settings.

Create the Agent with LLM:
    When creating an Agent instance, assign the default_llm to the llm parameter of the agent. This ensures that the agent uses the LM Studio model for processing language tasks.

example

    from crewai import Agent

from langchain_openai import ChatOpenAI

Other imports and setup...

Initialize the LLM with LM Studio settings

default_llm = ChatOpenAI(
openai_api_base="http://localhost:1234/v1",
openai_api_key="NA",
model_name="NA"
)

Create an Agent with the LLM

writer = Agent(
role='Tech Content Strategist',
goal='Craft compelling content on tech advancements',
backstory="""You are a renowned Content Strategist, known for
your insightful and engaging articles.
You transform complex concepts into compelling narratives.""",
verbose=True,
allow_delegation=True,
llm=default_llm # This is important
)

[KGuzikowski]

It no longer works. Does anyone know why?

[ViCtOr-dev13]

Update for 06 feb 2025 :

from crewai import Agent, Crew, Process, Task, LLM

lmstudio_llama = LLM(model="openai/llama-3-8b-instruct-64k", base_url="http://localhost:1234/v1", api_key="sk-1234")

@agent
def comment_extractor(self) -> Agent:
    return Agent(
        config=self.agents_config["comment_extractor"],
        llm=self.lmstudio_llama
    )

crewai==0.95.0