A self-hosted, offline, ChatGPT-like chatbot with different LLM support. 100% private, with no data leaving your device.

How to install

---

Install OpenLLM anywhere else with Docker

You can run OpenLLM on any x86 system. Make sure you have Docker installed.

Then, clone this repo and cd into it:

git clone https://github.com/edgar971/open-chat.git
cd open-chat

You can now run OpenLLM with any of the following models depending upon your hardware:

Model size	Model used	Minimum RAM required	How to start OpenLLM
7B	Nous Hermes Llama 2 7B (GGML q4_0)	8GB	docker compose up -d
13B	Nous Hermes Llama 2 13B (GGML q4_0)	16GB	docker compose -f docker-compose-13b.yml up -d
70B	Meta Llama 2 70B Chat (GGML q4_0)	48GB	docker compose -f docker-compose-70b.yml up -d

You can access OpenLLM at http://localhost:3000.

To stop OpenLLM, run:

docker compose down

---

API Configuration

Additional settings can be found here and added as env variables or arguments to the run.sh (--n_ctx 12) script.

Example:

version: '3'
services:
  api:
    image: ghcr.io/edgar971/open-chat-cuda:latest
    environment:
      - MODEL=/path/to/your/model
      - N_CTX=4096
    ports:

Acknowledgements

A massive thank you to the following developers and teams for making OpenLLM possible:

• Mckay Wrigley for building Chatbot UI.
• Georgi Gerganov for implementing llama.cpp.
• Andrei for building the Python bindings for llama.cpp.
• NousResearch for fine-tuning the Llama 2 7B and 13B models.
• Tom Jobbins for quantizing the Llama 2 models.
• Meta for releasing Llama 2 under a permissive license.