Skip to content

Commit

Permalink
docs: Ollama doc (#662)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Wendong-Fan
Wendong-Fan committed Jun 20, 2024
1 parent bfb9da0 commit d9e6f8d
Show file tree
Hide file tree
Showing 2 changed files with 110 additions and 87 deletions.
94 changes: 53 additions & 41 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -170,55 +170,67 @@ python examples/ai_society/role_playing.py
Please note that the environment variable is session-specific. If you open a new terminal window or tab, you will need to set the API key again in that new session.


## Use Open-Source Models as Backends
## Use Open-Source Models as Backends (ex. using Ollama to set Llama 3 locally)

The basic workflow of using an open-sourced model as the backend is based on an external server running LLM inference service, e.g. during the development we chose [FastChat](https://github.com/lm-sys/FastChat) to run the service.

We do not fix the choice of server to decouple the implementation of any specific LLM inference server with CAMEL (indicating the server needs to be deployed by the user himself). But the server to be deployed must satisfy that **it supports OpenAI-compatible APIs, especially the method `openai.ChatCompletion.create`**.

Here are some instructions for enabling open-source backends, where we use the [FastChat](https://github.com/lm-sys/FastChat) and a LLaMA2-based model ([`meta-llama/Llama-2-7b-chat-hf`](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)) in the example. Please install FastChat in advance following their installation guidance.

1. Before running CAMEL, we should firstly launch FastChat server following the guidance on https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md. The instructions summarized below should be kept running **in separate processes**:

```sh
# Launch the controller
python -m fastchat.serve.controller

# Launch the model worker(s)
python3 -m fastchat.serve.model_worker --model-path meta-llama/Llama-2-7b-chat-hf

# Launch the RESTful API server
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```
- Download [Ollama](https://ollama.com/download).
- After setting up Ollama, pull the Llama3 model by typing the following command into the terminal:
```bash
ollama pull llama3
```
- Create a ModelFile similar the one below in your project directory.
```bash
FROM llama3

2. After observing the controller successfully receiving the heart beat signal from the worker, the server should be ready for use at http://localhost:8000/v1.
# Set parameters
PARAMETER temperature 0.8
PARAMETER stop Result

3. Then we can try on running `role_playing_with_open_source_model.py`, where each agent in this example is initialized with specifying the `model_path` and `server_url`, similar to the example code below:
# Sets a custom system message to specify the behavior of the chat assistant

```python
system_message = # ...
# Leaving it blank for now.

model = ModelFactory.create(
model_platform=ModelPlatformType.OPENSOURCE
model_type=ModelType.LLAMA_2,
model_config=OpenSourceConfig(
model_path="meta-llama/Llama-2-7b-chat-hf",
server_url="http://localhost:8000/v1",
),
)
SYSTEM """ """
```
- Create a script to get the base model (llama3) and create a custom model using the ModelFile above. Save this as a .sh file:
```bash
#!/bin/zsh

agent = ChatAgent(
system_message,
model=model,
)
```
# variables
model_name="llama3"
custom_model_name="camel-llama3"

### Supported Models
#get the base model
ollama pull $model_name

- LLaMA2-based models
- example: [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)
- Vicuna-based models
- example: [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5)
#create the model file
ollama create $custom_model_name -f ./Llama3ModelFile
```
- Navigate to the directory where the script and ModelFile are located and run the script. Enjoy your Llama3 model, enhanced by CAMEL's excellent agents.
```python
from camel.agents import ChatAgent
from camel.messages import BaseMessage
from camel.models import ModelFactory
from camel.types import ModelPlatformType

ollama_model = ModelFactory.create(
model_platform=ModelPlatformType.OLLAMA,
model_type="llama3",
url="http://localhost:11434/v1",
model_config_dict={"temperature": 0.4},
)

assistant_sys_msg = BaseMessage.make_assistant_message(
role_name="Assistant",
content="You are a helpful assistant.",
)
agent = ChatAgent(assistant_sys_msg, model=ollama_model, token_limit=4096)

user_msg = BaseMessage.make_user_message(
role_name="User", content="Say hi to CAMEL"
)
assistant_response = agent.step(user_msg)
print(assistant_response.msg.content)
```

## Data (Hosted on Hugging Face)
| Dataset | Chat format | Instruction format | Chat format (translated) |
Expand Down
103 changes: 57 additions & 46 deletions docs/get_started/setup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,55 +112,66 @@ These commands on Windows will set the environment variable for the duration of


### [Option 2] Using Local Models
The high-level idea is to deploy a server with the local model in the backend and use it as a local drop-in replacement for the API. We here use [FastChat](https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md) as an example.
In the current landscape, for those seeking highly stable content generation, OpenAI's GPT-3.5 turbo, GPT-4o are often recommended. However, the field is rich with many other outstanding open-source models that also yield commendable results. CAMEL can support developers to delve into integrating these open-source large language models (LLMs) to achieve project outputs based on unique input ideas.

0. Install the FastChat package with the following command, or see [here](https://github.com/lm-sys/FastChat/tree/main#install) for other options.
```bash
pip3 install "fschat[model_worker,webui]"
```
#### Example: Using Ollama to set Llama 3 locally

1. Starting the FastChat server in the backend.
```python
# Launch the fastchat controller
python -m fastchat.serve.controller
- Download [Ollama](https://ollama.com/download).
- After setting up Ollama, pull the Llama3 model by typing the following command into the terminal:
```bash
ollama pull llama3
```
- Create a ModelFile similar the one below in your project directory.
```bash
FROM llama3

# Launch the model worker
python -m fastchat.serve.model_worker \
--model-path meta-llama/Llama-2-7b-chat-hf # a local folder or HuggingFace repo Name
# Set parameters
PARAMETER temperature 0.8
PARAMETER stop Result

# Launch the API server
python -m fastchat.serve.openai_api_server \
--host localhost \
--port 8000
```
# Sets a custom system message to specify the behavior of the chat assistant

# Leaving it blank for now.

2. Initialize the agent.
```python
# Import the necessary classes
from camel.configs import ChatGPTConfig, OpenSourceConfig
from camel.types import ModelType, ModelPlatformType
from camel.models import ModelFactory

# Set the LLM model type and model config
model_platform = ModelPlatformType.OPENSOURCE
model_type=ModelType.LLAMA_2 # Specify the model type
model_config=OpenSourceConfig(
model_path='meta-llama/Llama-2-7b-chat-hf', # a local folder or HuggingFace repo Name
server_url='http://localhost:8000/v1') # The url with the set port number

# Create the backend model
model = ModelFactory.create(
model_platform=model_platform,
model_type=model_type,
model_config=model_config)

# Set the arguments
agent_kwargs = dict(
model=model,
token_limit=2046, # [Optional] Choose the ideal limit
)

# Now your agent is ready to play
agent = ChatAgent(sys_msg, **agent_kwargs)
```
SYSTEM """ """
```
- Create a script to get the base model (llama3) and create a custom model using the ModelFile above. Save this as a .sh file:
```bash
#!/bin/zsh

# variables
model_name="llama3"
custom_model_name="camel-llama3"

#get the base model
ollama pull $model_name

#create the model file
ollama create $custom_model_name -f ./Llama3ModelFile
```
- Navigate to the directory where the script and ModelFile are located and run the script. Enjoy your Llama3 model, enhanced by CAMEL's excellent agents.
```python
from camel.agents import ChatAgent
from camel.messages import BaseMessage
from camel.models import ModelFactory
from camel.types import ModelPlatformType

ollama_model = ModelFactory.create(
model_platform=ModelPlatformType.OLLAMA,
model_type="llama3",
url="http://localhost:11434/v1",
model_config_dict={"temperature": 0.4},
)

assistant_sys_msg = BaseMessage.make_assistant_message(
role_name="Assistant",
content="You are a helpful assistant.",
)
agent = ChatAgent(assistant_sys_msg, model=ollama_model, token_limit=4096)

user_msg = BaseMessage.make_user_message(
role_name="User", content="Say hi to CAMEL"
)
assistant_response = agent.step(user_msg)
print(assistant_response.msg.content)
```

0 comments on commit d9e6f8d

Please sign in to comment.