ServerlessLLM Demo

Setup

1. Set model dir

export MODEL_FOLDER=/mnt/nvme0n1/lrq/

2. Use docker compose to setup the demo Use the following docker compose yaml file:

services:
  # Head Node
  sllm_head:
    build:
      context: ../../
      dockerfile: Dockerfile
    image: lrq619/sllm:latest
    container_name: sllm_head
    environment:
      - MODEL_FOLDER=${MODEL_FOLDER}
      - MODE=HEAD
    ports:
      - "6379:6379"    # Redis port
      - "8343:8343"    # ServerlessLLM port
    networks:
      - sllm_network
    command: ["--pushgateway_url", "pushgateway:9091"]
  
  pushgateway:
    image: prom/pushgateway
    container_name: prometheus_pushgateway
    ports:
      - "9091:9091"
    networks:
      - sllm_network

  # Worker Node 0
  sllm_worker_0:
    build:
      context: ../../
      dockerfile: Dockerfile
    image: lrq619/sllm:latest
    container_name: sllm_worker_0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: ["gpu"]
              device_ids: ["0","1","2","3"] # Assigns GPU 0 to the worker
    environment:
      - WORKER_ID=0
      - STORAGE_PATH=/models
      - MODE=WORKER
    networks:
      - sllm_network
    volumes:
      - ${MODEL_FOLDER}:/models
    command: ["--mem-pool-size", "16GB", "--registration-required", "true"] # Customize the memory pool size here

networks:
  sllm_network:
    driver: bridge
    name: sllm

Then executes docker compose -f examples/docker/demo-docker-compose.yml up -d 3. Register model

conda activate sllm
sllm-cli deploy --model "meta-llama/Llama-3.2-3B-Instruct"

4. Start prometheus

cd ~/proj/prometheus
./prometheus --config.file=prometheus.yml

Demo

1. Now, in the same terminal, open a chatbox application

cd ~/proj/demo/
conda activate demo
python chatbot.py

This will open a chatbot application, you can type in prompt and get response from ServerlessLLM(This would typically take ~30s) (Notice that response might not be very meaningful, llama3-8B is not a very stable model, just explain to the student that we use a small model and the accuracy is not very high if they ask) 4. On chrome, login to https://lrq619.grafana.net/dashboards It should be a window always opened there, if you encounter any problem, please contact Ruiqi asap You should be able to see two curves: With the yellow curve denoting the number of GPU, green curve denoting request arrival rate(request/minute) 5. Finally, on the vscode tunnel, start to send multiple requests at a very fast speed

cd ~/proj/demo/
conda activate demo
python chatbot.py

On the input, type cmd:send_requests, the script will start to send 100 requests to ServerlessLLM and the two curves will show something like this:

Clean up

Can just keep it running, or if you want, can also ctrl+c on the terminal you started the sllm-serve start --pushgateway_url localhost:9091 command

AMD demo

1. Login to hacchead node:

ssh lrq619@xacchead.d2.comp.nus.edu.sg

Then login to hacc-gpu4 node:

ssh lrq619@hacc-gpu4 -p 23

2. Start the singularity image using apptainer:

apptainer exec --rocm --bind /data/lrq619/models:/models /data/lrq619/sllm_store_rocm.sif /bin/bash

3. After enter apptainer, execute the following:

conda activate py_3.10
sllm-store start --storage-path /models > ./sllm_store.log 2>&1 &
# Wait for around 20 seconds, check if sllm-store starts
python load_vllm_model.py --model-name "facebook/opt-1.3b" --storage-path "/models"

4. Expected output:

Capturing CUDA graph shapes: 100%|██████████████████████████████████████████████████████████████| 35/35 [00:12<00:00,  2.70it/s]
INFO 03-27 14:50:36 model_runner.py:1549] Graph capturing finished in 13 secs, took 0.06 GiB
INFO 03-27 14:50:36 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 17.12 seconds
Processed prompts: 100%|███████████████████| 4/4 [00:00<00:00, 12.59it/s, est. speed input: 81.86 toks/s, output: 201.51 toks/s]
Prompt: 'Hello, my name is', Generated text: ' Joel, and I have been working as a web designer/developer for the'
Prompt: 'The president of the United States is', Generated text: " speaking in an increasingly important national security forum and he's not using the right words"
Prompt: 'The capital of France is', Generated text: " Paris.\nYeah but you couldn't get it through a French newspaper!"
Prompt: 'The future of AI is', Generated text: ' literally in your hands\nDespite all the hype, AI isn’t here'

5. Cleaning

ps -ef | grep sllm-store
kill -9 <pid>

6. Backup python script for load_vllm_model.py

import argparse
import os

from vllm import LLM, SamplingParams

parser = argparse.ArgumentParser(
    description="Save a model from HuggingFace model hub."
)
parser.add_argument(
    "--model-name",
    type=str,
    required=True,
    help="Model name from HuggingFace model hub.",
)
parser.add_argument(
    "--storage-path",
    type=str,
    default="./models",
    help="Local path to save the model.",
)

args = parser.parse_args()

model_name = args.model_name
storage_path = args.storage_path
model_path = os.path.join(storage_path, model_name)

llm = LLM(model=model_path, load_format="serverless_llm", dtype="float16", gpu_memory_utilization=0.3)

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")