Kubernetes Inference Platform

A production-grade ML inference platform on Kubernetes, built with KServe, NVIDIA GPU Operator, and observability via Prometheus/Grafana. Deployed cost-efficiently on Vultr Kubernetes Engine for portfolio validation, with portable manifests for AWS EKS and local kind clusters.

Layer Infrastructure

│ L5: Workload InferenceService (Model) │

│ L4: Platform KServe + Knative + GPU Operator │

│ L3: Observability Prometheus + Grafana + DCGM │

│ L2: Kubernetes VKE control plane + node pools │

│ L1: Cloud Vultr region (compute, network, LB) │

Platform Architecture

flowchart TB
    classDef user fill:#e1f5ff,stroke:#0277bd,stroke-width:2px,color:#000
    classDef aws fill:#ff9900,stroke:#232f3e,stroke-width:2px,color:#000
    classDef k8s fill:#326ce5,stroke:#1a3a8c,stroke-width:2px,color:#fff
    classDef ml fill:#7b1fa2,stroke:#4a148c,stroke-width:2px,color:#fff
    classDef obs fill:#43a047,stroke:#1b5e20,stroke-width:2px,color:#fff
    classDef app fill:#c62828,stroke:#8e0000,stroke-width:2px,color:#fff

    User["👤 User"]:::user

    subgraph AWS["☁️ AWS eu-central-1"]
        direction TB

        subgraph VPC["VPC 10.0.0.0/16"]
            direction TB

            subgraph SystemNodes["System Nodes (m5.large ×2, always-on)"]
                direction LR
                Karpenter["karpenter controller"]:::k8s
                KServeCtrl["kserve + knative controllers"]:::k8s
                CertMgr["cert-manager"]:::k8s
                LBCtrl["aws-lb-controller"]:::k8s
                GPUOp["gpu-operator"]:::k8s
            end

            subgraph LLMGatewayNS["ns: llm-gateway (PSS restricted)"]
                direction LR
                Gateway["llm-gateway\n:8000"]:::app
                Redis["redis (standalone,\nno persistence)"]:::app
                Gateway -->|cache lookup| Redis
            end

            subgraph ModelsNS["ns: models"]
                direction LR
                Small["qwen-0-5b-predictor\n1× GPU"]:::ml
                Large["qwen-7b-predictor\n1× GPU"]:::ml
            end

            subgraph GPUNodes["GPU Spot Nodes — Karpenter-provisioned on demand"]
                direction LR
                Node1["g5/g4dn/g6 node\nL4/T4 GPU"]
                Node2["g5/g4dn/g6 node\nL4/T4 GPU"]
            end

            NLB["NLB (internet-facing)\nGrafana only"]:::aws
        end
    end

    subgraph ObsStack["📊 Observability"]
        direction LR
        Prom["Prometheus"]:::obs
        Graf["Grafana"]:::obs
        DCGM["DCGM exporter"]:::obs
    end

    User -->|HTTP :8000| Gateway
    Gateway -->|cache miss| Small
    Gateway -->|cache miss| Large
    Small -.scheduled on.-> Node1
    Large -.scheduled on.-> Node2
    Karpenter -.provisions.-> Node1
    Karpenter -.provisions.-> Node2
    GPUOp -.labels nvidiacom/gpu.-> Node1
    GPUOp -.labels nvidiacom/gpu.-> Node2
    LBCtrl -.manages.-> NLB
    Graf -->|service type=LoadBalancer| NLB
    User -->|HTTPS| NLB
    Prom -.scrapes.-> Small
    Prom -.scrapes.-> Large
    Prom -.scrapes.-> Gateway
    DCGM -.GPU metrics.-> Node1
    DCGM -.GPU metrics.-> Node2
    Prom --> Graf

    NetPolicy["🔒 NetworkPolicy: only llm-gateway\n+ monitoring ns can reach models ns"]
    NetPolicy -.enforces.-> ModelsNS

Loading

Why two deployment modes

qwen-0-5b runs in Serverless mode (Knative-backed, serving.kserve.io/deploymentMode: Serverless annotation) and scales to zero GPU nodes after idle traffic — cheap to leave running, at the cost of a multi-minute cold start (node provisioning + image pull + model load) on the first request after idle.

qwen-7b runs in the cluster's default RawDeployment mode — a static, always-warm replica with no cold start, at the cost of a continuously billed GPU node.

This is a deliberate split, not an oversight: it mirrors how production inference platforms commonly treat low-traffic/experimental models versus latency-sensitive ones, and gives a concrete cold-start-vs-warm-latency comparison to point to.