Skip to content

README.md

Latest commit

 

History

History
82 lines (59 loc) · 2.52 KB

README.md

File metadata and controls

82 lines (59 loc) · 2.52 KB
Raw

LLM Stack

A helm chart for a self-hosted llm stack featuring:

  • Apisix as the AI gateway
    • running in api-based standalone mode (allowing only full config updates via api)
    • having a config seeder job that pushes the route and consumer configs initially using a headless service for pod discovery
  • An optional VLLM deployment for hosting an LLM in the cluster
  • Optional prometheus and grafana components for local development
  • values.yaml files and basic request scripts for local development in examples/ and bin/

chart overview

Local Development

Minimum requirements for local vllm

  • 48gb RAM (64gb would be better)

Creating the cluster

k3d cluster create llm-stack-dev --port "80:80@loadbalancer"
kubectl create namespace llm-stack

Creating the Secrets

For the apisix initial config:

kubectl create secret generic llm-stack-apisix-initial-config-secret --from-literal=provider_api_key='abc' --from-literal=consumers='[{"name": "consumerA", "key": "sk-client-v1-abcdef123456"}]' -n llm-stack
  • provider_api_key is the key of the provider you are forwarding the requests to, e.g. scaleway, can be anything when using vllm
  • consumers is a stringified json array of consumers

For the apisix admin API:

kubectl create secret generic llm-stack-apisix-admin-secret --from-literal=admin='abc' --from-literal=viewer='def' -n llm-stack
  • admin is the admin key for editing
  • viewer is the admin key for viewing only

Installing the chart

With cpu-based VLLM and a tiny model enabled

helm install dev-release . -f examples/local-vllm-cpu.yaml -n llm-stack

Wait for the vllm liveness probe to start (~4 min) and make a test request:

bash bin/test-request-local-vllm.sh

Using scaleway instead of vllm

Adjust the scaleway project id in the apisixInitialConfig.base_url in examples/scaleway.yaml with your own credentials.

helm install dev-release . -f examples/scaleway.yaml -n llm-stack
bash bin/test-request-scaleway.sh

Observability

Grafana

Log into grafana at http://grafana.localhost using the default credentials (username: admin, password: admin)

Useful queries:

Successful requests for apisix grouped by consumer at 1 min interval:

sum(increase(apisix_http_status{code=~"[2].."}[1m])) by (consumer)

Failing requests for apisix grouped by consumer at 1 min interval:

sum(increase(apisix_http_status{code=~"[45].."}[1m])) by (consumer)