GPU AI Platform on MicroK8s with vLLM, RAG and LGTM Observability

A complete self-hosted AI platform deployed on AWS EC2 using MicroK8s, NVIDIA GPU Operator, vLLM, Open WebUI, Qdrant, FastAPI-based RAG, and a full LGTM observability stack (Grafana, Prometheus, Loki, Tempo, OpenTelemetry).

---

Project Overview

This project demonstrates how to build a production-style AI platform on Kubernetes using a single GPU-enabled EC2 instance.

The platform includes:

• GPU-enabled Kubernetes cluster using MicroK8s
• NVIDIA GPU Operator integration
• vLLM model serving
• Qwen2.5-1.5B-Instruct model
• Open WebUI chat interface
• Qdrant vector database
• Custom Retrieval-Augmented Generation (RAG) application
• Prometheus metrics
• Grafana dashboards
• Loki log aggregation
• Tempo distributed tracing
• OpenTelemetry Collector

The deployment was executed on:

AWS EC2 g5.xlarge
Ubuntu 24.04 LTS
NVIDIA A10G GPU (24 GB VRAM)
MicroK8s Kubernetes

---

Architecture

                    ┌─────────────────┐
                    │   Open WebUI    │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │      vLLM       │
                    │ Qwen2.5-1.5B    │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │     RAG API     │
                    │    FastAPI      │
                    └────────┬────────┘
                             │
           ┌─────────────────┴─────────────────┐
           │                                   │
           ▼                                   ▼
┌────────────────────┐               ┌────────────────────┐
│       Qdrant       │               │ SentenceTransformer│
│    Vector Store    │               │     Embeddings     │
└────────────────────┘               └────────────────────┘


                Observability Stack

┌───────────────────────────────────────────────┐
│                    Grafana                    │
└───────────────┬─────────────┬─────────────────┘
                │             │
                ▼             ▼
        Prometheus          Loki
         Metrics            Logs
                │             │
                └──────┬──────┘
                       ▼
                     Tempo
                     Traces
                       │
                       ▼
            OpenTelemetry Collector

---

Technology Stack

Infrastructure

• AWS EC2 g5.xlarge
• Ubuntu 24.04 LTS
• NVIDIA A10G GPU

Container Platform

• Kubernetes
• MicroK8s
• Containerd
• Helm

AI Components

• vLLM
• Qwen2.5-1.5B-Instruct
• Open WebUI
• FastAPI
• Qdrant
• Sentence Transformers

Observability

• Grafana
• Prometheus
• Loki
• Promtail
• Tempo
• OpenTelemetry Collector

---

Repository Structure

.
├── README.md
├── docs/
├── manifests/
│   ├── vllm/
│   ├── qdrant/
│   ├── openwebui/
│   ├── rag-app/
│   └── observability/
│
├── applications/
│   └── rag-app/
│       ├── Dockerfile
│       ├── main.py
│       └── requirements.txt
│
├── runbooks/
├── scripts/
├── screenshots/
└── architecture/

---

Features

GPU Inference

• GPU-enabled Kubernetes scheduling
• NVIDIA GPU Operator
• CUDA validation
• vLLM GPU inference

LLM Serving

• OpenAI-compatible API
• vLLM deployment
• Qwen2.5 model serving
• Open WebUI integration

Retrieval-Augmented Generation

Document Ingestion

POST /ingest

Semantic Search

POST /query

Health Check

GET /health

Vector Database

• Qdrant
• Dense embeddings
• Similarity search

Observability

Metrics

• Node Metrics
• Kubernetes Metrics
• AI Service Metrics

Logs

• Loki
• Promtail
• Grafana Explore

Traces

• Tempo
• OpenTelemetry Collector
• Distributed Tracing

---

Validation Performed

Successfully validated:

• NVIDIA Driver Installation
• GPU Scheduling
• CUDA Test Pod
• vLLM Startup
• OpenAI Compatible Endpoint
• Open WebUI Integration
• Qdrant Deployment
• RAG Ingestion
• RAG Querying
• Prometheus Metrics
• Loki Logs
• Tempo Traces
• OpenTelemetry Collector

---

Sample RAG Query

Request:

{
  "question": "What is Qdrant used for?"
}

Response:

{
  "answer": "...",
  "sources": [
    {
      "text": "Qdrant is a vector database used for retrieval augmented generation."
    }
  ]
}

---

Resource Utilization

Final deployment resource usage:

CPU Usage      ~7%
Memory Usage   ~49%

Running successfully on:

4 vCPU
16 GB RAM
1 x NVIDIA A10G GPU

---

Challenges Solved

vLLM Kubernetes Service Variable Conflict

Issue:

VLLM_PORT appears to be a URI

Solution:

enableServiceLinks: false

---

Open WebUI Context Length Errors

Issue:

Maximum context length exceeded

Solution:

--max-model-len 8192

---

Qdrant Client Compatibility

Issue:

AttributeError:
QdrantClient has no attribute search

Solution:

query_points()

---

Kubernetes CPU Scheduling

Issue:

Insufficient CPU

Solution:

Optimized CPU requests across:

• vLLM
• Open WebUI
• Qdrant
• Prometheus

---

Screenshots

Add screenshots here:

screenshots/
├── open-webui.png
├── grafana-dashboard.png
├── loki-logs.png
├── tempo-traces.png
├── qdrant-health.png
└── vllm-api-test.png

---

Future Enhancements

• Multi-node Kubernetes deployment
• Larger LLM models
• K6 load testing
• GitOps deployment using ArgoCD
• CI/CD integration
• GPU autoscaling
• Multi-tenant model serving

---

Skills Demonstrated

• Kubernetes
• MicroK8s
• NVIDIA GPU Operator
• AWS EC2
• vLLM
• Open WebUI
• Qdrant
• RAG
• FastAPI
• Docker
• Helm
• Prometheus
• Grafana
• Loki
• Tempo
• OpenTelemetry
• Observability
• Platform Engineering
• Site Reliability Engineering

---

Author

Deepak Deorari

DevOps-III (Site Reliability Engineer III)

Built as a hands-on AI Platform Engineering, Kubernetes, GPU Infrastructure and Observability project.