🦾 OpenLLM: Self-Hosting LLMs Made Easy

📖 Introduction

OpenLLM helps developers run any open-source LLMs, such as Llama 2 and Mistral, as OpenAI-compatible API endpoints, locally and in the cloud, optimized for serving throughput and production deployment.

• 🚂 Support a wide range of open-source LLMs including LLMs fine-tuned with your own data
• ⛓️ OpenAI compatible API endpoints for seamless transition from your LLM app to open-source LLMs
• 🔥 State-of-the-art serving and inference performance
• 🎯 Simplified cloud deployment via BentoML

💾 TL/DR

For starter, we provide two ways to quickly try out OpenLLM:

Jupyter Notebooks

Try this OpenLLM tutorial in Google Colab: Serving Phi 3 with OpenLLM.

🏃 Get started

The following provides instructions for how to get started with OpenLLM locally.

Prerequisites

You have installed Python 3.9 (or later) and pip. We highly recommend using a Virtual Environment to prevent package conflicts.

Install OpenLLM

Install OpenLLM by using pip as follows:

pip install openllm

To verify the installation, run:

$ openllm -h

Start a LLM server

OpenLLM allows you to quickly spin up an LLM server using openllm start. For example, to start a Phi-3 server, run the following:

openllm start microsoft/Phi-3-mini-4k-instruct --trust-remote-code

To interact with the server, you can visit the web UI at http://0.0.0.0:3000/ or send a request using curl. You can also use OpenLLM’s built-in Python client to interact with the server:

import openllm

client = openllm.HTTPClient('http://localhost:3000')
client.generate('Explain to me the difference between "further" and "farther"')

OpenLLM seamlessly supports many models and their variants. You can specify different variants of the model to be served. For example:

openllm start <model_id> --<options>

🧩 Supported models

OpenLLM currently supports the following models. By default, OpenLLM doesn't include dependencies to run all models. The extra model-specific dependencies can be installed with the instructions below.

Baichuan

Quickstart

Note: Baichuan requires to install with:

pip install "openllm[baichuan]"

Run the following command to quickly spin up a Baichuan server:

TRUST_REMOTE_CODE=True openllm start baichuan-inc/baichuan-7b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Baichuan variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Baichuan-compatible models.

Supported models

You can specify any of the following Baichuan models via openllm start:

• baichuan-inc/baichuan2-7b-base
• baichuan-inc/baichuan2-7b-chat
• baichuan-inc/baichuan2-13b-base
• baichuan-inc/baichuan2-13b-chat

ChatGLM

Quickstart

Note: ChatGLM requires to install with:

pip install "openllm[chatglm]"

Run the following command to quickly spin up a ChatGLM server:

TRUST_REMOTE_CODE=True openllm start thudm/chatglm-6b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any ChatGLM variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more ChatGLM-compatible models.

Supported models

You can specify any of the following ChatGLM models via openllm start:

• thudm/chatglm-6b
• thudm/chatglm-6b-int8
• thudm/chatglm-6b-int4
• thudm/chatglm2-6b
• thudm/chatglm2-6b-int4
• thudm/chatglm3-6b

Dbrx

Quickstart

Note: Dbrx requires to install with:

pip install "openllm[dbrx]"

Run the following command to quickly spin up a Dbrx server:

TRUST_REMOTE_CODE=True openllm start databricks/dbrx-instruct

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Dbrx variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Dbrx-compatible models.

Supported models

You can specify any of the following Dbrx models via openllm start:

• databricks/dbrx-instruct
• databricks/dbrx-base

DollyV2

Quickstart

Run the following command to quickly spin up a DollyV2 server:

TRUST_REMOTE_CODE=True openllm start databricks/dolly-v2-3b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any DollyV2 variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more DollyV2-compatible models.

Supported models

You can specify any of the following DollyV2 models via openllm start:

• databricks/dolly-v2-3b
• databricks/dolly-v2-7b
• databricks/dolly-v2-12b

Falcon

Quickstart

Note: Falcon requires to install with:

pip install "openllm[falcon]"

Run the following command to quickly spin up a Falcon server:

TRUST_REMOTE_CODE=True openllm start tiiuae/falcon-7b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Falcon variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Falcon-compatible models.

Supported models

You can specify any of the following Falcon models via openllm start:

• tiiuae/falcon-7b
• tiiuae/falcon-40b
• tiiuae/falcon-7b-instruct
• tiiuae/falcon-40b-instruct

Gemma

Quickstart

Note: Gemma requires to install with:

pip install "openllm[gemma]"

Run the following command to quickly spin up a Gemma server:

TRUST_REMOTE_CODE=True openllm start google/gemma-7b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Gemma variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Gemma-compatible models.

Supported models

You can specify any of the following Gemma models via openllm start:

• google/gemma-7b
• google/gemma-7b-it
• google/gemma-2b
• google/gemma-2b-it

GPTNeoX

Quickstart

Run the following command to quickly spin up a GPTNeoX server:

TRUST_REMOTE_CODE=True openllm start eleutherai/gpt-neox-20b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any GPTNeoX variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more GPTNeoX-compatible models.

Supported models

You can specify any of the following GPTNeoX models via openllm start:

• eleutherai/gpt-neox-20b

Llama

Quickstart

Note: Llama requires to install with:

pip install "openllm[llama]"

Run the following command to quickly spin up a Llama server:

TRUST_REMOTE_CODE=True openllm start NousResearch/llama-2-7b-hf

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Llama variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Llama-compatible models.

Supported models

You can specify any of the following Llama models via openllm start:

• meta-llama/Llama-2-70b-chat-hf
• meta-llama/Llama-2-13b-chat-hf
• meta-llama/Llama-2-7b-chat-hf
• meta-llama/Llama-2-70b-hf
• meta-llama/Llama-2-13b-hf
• meta-llama/Llama-2-7b-hf
• NousResearch/llama-2-70b-chat-hf
• NousResearch/llama-2-13b-chat-hf
• NousResearch/llama-2-7b-chat-hf
• NousResearch/llama-2-70b-hf
• NousResearch/llama-2-13b-hf
• NousResearch/llama-2-7b-hf

Mistral

Quickstart

Note: Mistral requires to install with:

pip install "openllm[mistral]"

Run the following command to quickly spin up a Mistral server:

TRUST_REMOTE_CODE=True openllm start mistralai/Mistral-7B-Instruct-v0.1

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Mistral variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Mistral-compatible models.

Supported models

You can specify any of the following Mistral models via openllm start:

• HuggingFaceH4/zephyr-7b-alpha
• HuggingFaceH4/zephyr-7b-beta
• mistralai/Mistral-7B-Instruct-v0.2
• mistralai/Mistral-7B-Instruct-v0.1
• mistralai/Mistral-7B-v0.1

Mixtral

Quickstart

Note: Mixtral requires to install with:

pip install "openllm[mixtral]"

Run the following command to quickly spin up a Mixtral server:

TRUST_REMOTE_CODE=True openllm start mistralai/Mixtral-8x7B-Instruct-v0.1

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Mixtral variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Mixtral-compatible models.

Supported models

You can specify any of the following Mixtral models via openllm start:

• mistralai/Mixtral-8x7B-Instruct-v0.1
• mistralai/Mixtral-8x7B-v0.1

MPT

Quickstart

Note: MPT requires to install with:

pip install "openllm[mpt]"

Run the following command to quickly spin up a MPT server:

TRUST_REMOTE_CODE=True openllm start mosaicml/mpt-7b-instruct

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any MPT variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more MPT-compatible models.

Supported models

You can specify any of the following MPT models via openllm start:

• mosaicml/mpt-7b
• mosaicml/mpt-7b-instruct
• mosaicml/mpt-7b-chat
• mosaicml/mpt-7b-storywriter
• mosaicml/mpt-30b
• mosaicml/mpt-30b-instruct
• mosaicml/mpt-30b-chat

OPT

Quickstart

Note: OPT requires to install with:

pip install "openllm[opt]"

Run the following command to quickly spin up a OPT server:

openllm start facebook/opt-1.3b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any OPT variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more OPT-compatible models.

Supported models

You can specify any of the following OPT models via openllm start:

• facebook/opt-125m
• facebook/opt-350m
• facebook/opt-1.3b
• facebook/opt-2.7b
• facebook/opt-6.7b
• facebook/opt-66b

Phi

Quickstart

Note: Phi requires to install with:

pip install "openllm[phi]"

Run the following command to quickly spin up a Phi server:

TRUST_REMOTE_CODE=True openllm start microsoft/Phi-3-mini-4k-instruct

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Phi variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Phi-compatible models.

Supported models

You can specify any of the following Phi models via openllm start:

• microsoft/Phi-3-mini-4k-instruct
• microsoft/Phi-3-mini-128k-instruct
• microsoft/Phi-3-small-8k-instruct
• microsoft/Phi-3-small-128k-instruct
• microsoft/Phi-3-medium-4k-instruct
• microsoft/Phi-3-medium-128k-instruct

Qwen

Quickstart

Note: Qwen requires to install with:

pip install "openllm[qwen]"

Run the following command to quickly spin up a Qwen server:

TRUST_REMOTE_CODE=True openllm start qwen/Qwen-7B-Chat

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Qwen variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Qwen-compatible models.

Supported models

You can specify any of the following Qwen models via openllm start:

• qwen/Qwen-7B-Chat
• qwen/Qwen-7B-Chat-Int8
• qwen/Qwen-7B-Chat-Int4
• qwen/Qwen-14B-Chat
• qwen/Qwen-14B-Chat-Int8
• qwen/Qwen-14B-Chat-Int4

StableLM

Quickstart

Note: StableLM requires to install with:

pip install "openllm[stablelm]"

Run the following command to quickly spin up a StableLM server:

TRUST_REMOTE_CODE=True openllm start stabilityai/stablelm-tuned-alpha-3b

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any StableLM variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more StableLM-compatible models.

Supported models

You can specify any of the following StableLM models via openllm start:

• stabilityai/stablelm-tuned-alpha-3b
• stabilityai/stablelm-tuned-alpha-7b
• stabilityai/stablelm-base-alpha-3b
• stabilityai/stablelm-base-alpha-7b

StarCoder

Quickstart

Note: StarCoder requires to install with:

pip install "openllm[starcoder]"

Run the following command to quickly spin up a StarCoder server:

TRUST_REMOTE_CODE=True openllm start bigcode/starcoder

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any StarCoder variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more StarCoder-compatible models.

Supported models

You can specify any of the following StarCoder models via openllm start:

• bigcode/starcoder
• bigcode/starcoderbase

Yi

Quickstart

Note: Yi requires to install with:

pip install "openllm[yi]"

Run the following command to quickly spin up a Yi server:

TRUST_REMOTE_CODE=True openllm start 01-ai/Yi-6B

In a different terminal, run the following command to interact with the server:

export OPENLLM_ENDPOINT=http://localhost:3000
openllm query 'What are large language models?'

Note: Any Yi variants can be deployed with OpenLLM. Visit the HuggingFace Model Hub to see more Yi-compatible models.

Supported models

You can specify any of the following Yi models via openllm start:

• 01-ai/Yi-6B
• 01-ai/Yi-34B
• 01-ai/Yi-6B-200K
• 01-ai/Yi-34B-200K

More models will be integrated with OpenLLM and we welcome your contributions if you want to incorporate your custom LLMs into the ecosystem. Check out Adding a New Model Guide to learn more.

📐 Quantization

Quantization is a technique to reduce the storage and computation requirements for machine learning models, particularly during inference. By approximating floating-point numbers as integers (quantized values), quantization allows for faster computations, reduced memory footprint, and can make it feasible to deploy large models on resource-constrained devices.

OpenLLM supports the following quantization techniques

• LLM.int8(): 8-bit Matrix Multiplication through bitsandbytes
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression through bitsandbytes
• AWQ: Activation-aware Weight Quantization,
• GPTQ: Accurate Post-Training Quantization
• SqueezeLLM: Dense-and-Sparse Quantization.

Note

Make sure to use pre-quantized models weights when using with openllm start.

⚙️ Integrations

OpenLLM is not just a standalone product; it's a building block designed to integrate with other powerful tools easily. We currently offer integration with OpenAI's Compatible Endpoints, LlamaIndex, LangChain, and Transformers Agents.

OpenAI Compatible Endpoints

OpenLLM Server can be used as a drop-in replacement for OpenAI's API. Simply specify the base_url to llm-endpoint/v1 and you are good to go:

import openai

client = openai.OpenAI(base_url='http://localhost:3000/v1', api_key='na')  # Here the server is running on 0.0.0.0:3000

completions = client.chat.completions.create(
  prompt='Write me a tag line for an ice cream shop.', model=model, max_tokens=64, stream=stream
)

The compatible endpoints supports /completions, /chat/completions, and /models

Note

You can find out OpenAI example clients under the examples folder.

LlamaIndex

You can use llama_index.llms.openllm.OpenLLMAPI to interact with a LLM running server:

from llama_index.llms.openllm import OpenLLMAPI

Note

All synchronous and asynchronous API from llama_index.llms.LLM are supported.

LangChain

Spin up an OpenLLM server, and connect to it by specifying its URL:

from langchain.llms import OpenLLM

llm = OpenLLM(server_url='http://44.23.123.1:3000', server_type='http')
llm('What is the difference between a duck and a goose? And why there are so many Goose in Canada?')

🚀 Deploying models to production

There are several ways to deploy your LLMs:

🐳 Docker container

1. Building a Bento: With OpenLLM, you can easily build a Bento for a specific model, like mistralai/Mistral-7B-Instruct-v0.1, using the build command.:

   openllm build mistralai/Mistral-7B-Instruct-v0.1

   A Bento, in BentoML, is the unit of distribution. It packages your program's source code, models, files, artefacts, and dependencies.

2. Containerize your Bento

   bentoml containerize <name:version>

   This generates a OCI-compatible docker image that can be deployed anywhere docker runs. For best scalability and reliability of your LLM service in production, we recommend deploy with BentoCloud。

☁️ BentoCloud

Deploy OpenLLM with BentoCloud, the inference platform for fast moving AI teams.

1. Create a BentoCloud account: sign up here

2. Log into your BentoCloud account:

   bentoml cloud login --api-token <your-api-token> --endpoint <bento-cloud-endpoint>

Note

Replace <your-api-token> and <bento-cloud-endpoint> with your specific API token and the BentoCloud endpoint respectively.

3. Bulding a Bento: With OpenLLM, you can easily build a Bento for a specific model, such as mistralai/Mistral-7B-Instruct-v0.1:

   openllm build mistralai/Mistral-7B-Instruct-v0.1

4. Pushing a Bento: Push your freshly-built Bento service to BentoCloud via the push command:

   bentoml push <name:version>

5. Deploying a Bento: Deploy your LLMs to BentoCloud with a single bentoml deployment create command following the deployment instructions.

👥 Community

Engage with like-minded individuals passionate about LLMs, AI, and more on our Discord!

OpenLLM is actively maintained by the BentoML team. Feel free to reach out and join us in our pursuit to make LLMs more accessible and easy to use 👉 Join our Slack community!

🎁 Contributing

We welcome contributions! If you're interested in enhancing OpenLLM's capabilities or have any questions, don't hesitate to reach out in our discord channel.

Checkout our Developer Guide if you wish to contribute to OpenLLM's codebase.

📔 Citation

If you use OpenLLM in your research, we provide a citation to use:

@software{Pham_OpenLLM_Operating_LLMs_2023,
author = {Pham, Aaron and Yang, Chaoyu and Sheng, Sean and  Zhao, Shenyang and Lee, Sauyon and Jiang, Bo and Dong, Fog and Guan, Xipeng and Ming, Frost},
license = {Apache-2.0},
month = jun,
title = {{OpenLLM: Operating LLMs in production}},
url = {https://github.com/bentoml/OpenLLM},
year = {2023}
}