LLM Inference Service v1.0

Table of Contents

1. Overview
2. v1.0 Release Notes
3. Architecture
4. Features
5. API Reference
6. Deployment
7. GPU Monitoring
8. Quick Start
9. License
- 9.1. Third-Party Licenses
10. Support
11. Acknowledgments

1. Overview

The LLM Inference Service is a cost-effective, production-ready inference server that democratizes access to large language model deployment using consumer-grade hardware. Built to address the economic and technical challenges of modern LLM infrastructure, it provides a unified solution for teams who need enterprise capabilities without datacenter GPU costs.

1.1. Why This Project Exists

Economic Reality: The current LLM infrastructure landscape forces an impossible choice - either pay $30,000+ for datacenter GPUs (H100, A100) or struggle with fragmented, limited tooling. Most teams, researchers, and startups simply cannot justify the cost of datacenter hardware for experimentation and development.

Technical Gaps: Existing solutions have critical limitations:

Ollama: Limited enterprise features, poor GPU sharding, no comprehensive monitoring
vLLM: Complex multi-GPU quantization setup, steep learning curve
OpenAI API: Expensive, vendor lock-in, no control over infrastructure
No unified dashboard: No single interface for monitoring, debugging, and managing multiple models

Infrastructure Challenges: Teams need to maximize their existing hardware investment through intelligent utilization of motherboard PCIe lanes and consumer GPUs, rather than splurging on single high-VRAM datacenter cards.

1.2. What This Solution Provides

Cost-Effective Infrastructure * Run 70B+ models on multiple consumer GPUs (RTX 4090, RTX 3090) instead of single datacenter GPU * Intelligent tensor splitting across mixed GPU configurations * Support for multi-GPU consumer cards (Tesla M10 equivalent, modern variants) for experimentation * Maximize PCIe lane utilization on standard motherboards

Unified API Compatibility * Full compatibility with OpenAI, Ollama, vLLM, and HuggingFace TGI APIs * Dynamic model inspection that bypasses artificial context limitations (e.g., 32K models showing as 4K) * Think tag preservation for reasoning models * Real-time streaming with progress tracking

Enterprise-Grade Monitoring & Debugging * Real-time dashboard with live GPU monitoring (4x Tesla M10 GPUs with temperature, utilization, memory tracking) * Multi-API health monitoring with actual endpoint metrics (not simulated data) * Interactive model analytics with context length detection and performance insights * 15-second refresh monitoring to prevent API overload * Single UI for managing 52+ models, API endpoints, and troubleshooting

Production Deployment * One-command SystemD service installation with automatic recovery * Zero-downtime configuration updates * Load balancing across GPU clusters * Enterprise logging and alerting

Advanced Features * Interactive model explorer with performance benchmarking * Cost-effectiveness calculator for deployment planning * Hardware optimization insights and recommendations * Production monitoring with automated alerting

1.3. How It Works

Smart GPU Utilization The service automatically detects and optimally distributes workloads across available consumer GPUs, regardless of VRAM differences or generations. This enables teams to build powerful inference clusters using affordable consumer hardware instead of expensive datacenter solutions.

Dynamic Model Intelligence By integrating directly with Ollama CLI, the service dynamically discovers true model capabilities - solving the common problem where models artificially report 4K context limits when they actually support 32K+ tokens.

Unified Architecture A clean, modular codebase with format-specific adapters ensures compatibility across multiple AI providers while maintaining a single codebase and deployment model.

Real-World Impact: This approach enables startups and research teams to experiment with state-of-the-art models using existing hardware, while providing the monitoring and debugging capabilities needed for production deployment.

2. v1.0 Release Notes

2.1. What’s New in v1.0

The v1.0 release represents a production-ready milestone with comprehensive API compatibility, robust monitoring, and advanced hardware optimization features.

2.1.1. Major Features

🚀 Universal API Compatibility * Full support for OpenAI, Ollama, vLLM, and HuggingFace TGI APIs in a single service * Automatic smallest model selection when no model is specified across all endpoints * Seamless format adaptation with zero client-side changes required * Real-time API health monitoring with actual performance metrics (not simulated data)

📊 Advanced Dashboard System * 6 specialized dashboard pages with 15-second auto-refresh intervals * Real-time GPU monitoring with thermal management and power tracking * Interactive model analytics with true context detection (up to 131K+ tokens) * Hardware optimization insights with intelligent scoring (current: 100.0/100) * Dynamic configuration management with live updates

🔧 Production-Ready Features * Robust error handling and graceful degradation * SystemD service integration with automatic recovery * Comprehensive logging and structured debugging * Multi-GPU tensor splitting with optimal resource allocation * Zero-downtime configuration updates

💡 Intelligent Model Management * Dynamic model inspection bypassing artificial context limitations * 52+ model support with automatic capability detection * Think tag preservation for reasoning models (phi4-reasoning, deepseek-r1) * Performance benchmarking and optimization recommendations

2.1.2. Key Improvements

API Reliability * Fixed JavaScript dashboard errors with robust error handling * Enhanced chart initialization with proper validation * Improved request tracking with accurate duration calculations * Real-time metrics collection with proper endpoint mapping

Hardware Optimization * Optimized 4x Tesla M10 GPU configuration (25% tensor split each) * Thermal monitoring with intelligent alert thresholds (39-72°C range) * Memory efficiency tracking (28.5% utilization with scaling headroom) * Power consumption optimization (25-34W per GPU, 47-65% of limit)

Developer Experience * Comprehensive API documentation with real examples * Detailed troubleshooting guides and best practices * Production deployment checklists and monitoring setup * Complete Docker and SystemD deployment options

2.1.3. Technical Achievements

Zero Configuration: Works out-of-the-box with automatic model detection
Multi-API Support: Single codebase supporting 4 major AI API formats
Real-Time Monitoring: Live dashboard updates without API overload
Consumer GPU Focus: Optimized for RTX 4090/4080 and Tesla M10 configurations
Production Scalability: Designed for enterprise deployment patterns

2.2. Deployment Status

Component	Status	Details
Core API Server	✅ Production Ready	Flask-based with multi-format request adapters
Multi-GPU Support	✅ Production Ready	4x Tesla M10 with optimized tensor splitting
Dashboard System	✅ Production Ready	6 specialized dashboards, real-time monitoring
Model Management	✅ Production Ready	52+ models, context detection up to 131K tokens
API Compatibility	✅ Production Ready	OpenAI, Ollama, vLLM, HuggingFace TGI support
Hardware Optimization	✅ Production Ready	Intelligent recommendations, 100.0/100 scoring
Production Monitoring	✅ Production Ready	Health scoring, alerting, performance tracking

2.3. Getting Started with v1.0

Quick Installation

# Clone and install
git clone https://github.com/your-org/llm-inference-service.git
cd llm-inference-service

# Install as user service
./install-user-service.sh

# Start the service
systemctl --user start llm-inference

# Access dashboard
open http://localhost:11435/dashboard

Test All API Formats

# Test OpenAI compatibility (no model specified - uses smallest automatically)
curl -X POST http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}]}'

# Test Ollama compatibility
curl -X POST http://localhost:11435/api/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello"}'

# Test vLLM compatibility
curl -X POST http://localhost:11435/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello"}'

# Test HuggingFace TGI compatibility
curl -X POST http://localhost:11435/generate \
  -H "Content-Type: application/json" \
  -d '{"inputs": "Hello"}'

2.4. Migration from Beta

Existing installations can upgrade to v1.0 by:

Backup Configuration: cp -r config config.backup
Pull Latest Code: git pull origin main
Restart Service: systemctl --user restart llm-inference
Verify Dashboard: Visit http://localhost:11435/dashboard

No breaking changes - all existing configurations remain compatible.

3. Architecture

The LLM Inference Service follows a modular architecture with clear separation of concerns:

3.1. Core Components

Figure 1. System Architecture Overview

The service is organized into distinct layers:

API Gateway Layer: Handles incoming requests and routing
Request Processing: Manages format adapters and request tracking
Inference Engine: Core LLM execution with GPU management
Monitoring & Analytics: Real-time monitoring and metrics collection
Utilities: Supporting services for optimization and cost analysis

3.2. Component Interactions

Figure 2. Data Flow Through System Components

The request processing flow follows these key stages:

Client Request: Incoming HTTP request in various API formats
API Gateway: Routes and validates requests
Format Adaptation: Converts to internal format
Model Management: Loads model metadata and validates parameters
GPU Allocation: Distributes workload across available GPUs
Inference Execution: Runs LLAMA.cpp with tensor splitting
Response Processing: Formats output for client requirements

3.3. Deployment Architecture

Figure 3. Production Deployment Architecture

The deployment follows a production-ready pattern:

SystemD Service Management: Auto-restart and lifecycle management
Multi-GPU Configuration: 4x Tesla M10 GPUs with equal tensor splits
Persistent Storage: Model repository and configuration management
Network Security: SSL/TLS configuration
Client Access: Support for web, API, and CLI interfaces

3.4. Technical Design Patterns

3.4.1. Adapter Pattern

Used for multi-API compatibility, allowing seamless support for different AI service formats.

3.4.2. Observer Pattern

Implemented in monitoring systems for real-time metric collection and alerting.

3.4.3. Strategy Pattern

Applied in model selection and execution strategies based on workload characteristics.

3.4.4. Factory Pattern

Used for creating appropriate adapters and executors based on request types.

3.5. Component Details

3.5.1. API Gateway Layer

Flask API Server: Main HTTP server handling all requests
Route Handlers: URL routing and endpoint management
Request Validation: Input validation and security checks

3.5.2. Request Processing

Request Adapters: Format-specific request transformers
Model Manager: Model discovery and metadata management
Request Tracker: Active request monitoring and progress tracking

3.5.3. Inference Engine

LLAMA Executor: Core inference execution using llama.cpp
GPU Manager: Multi-GPU resource allocation and monitoring
Response Processor: Output formatting and post-processing

3.5.4. Monitoring & Analytics

GPU Monitor: Real-time GPU utilization and thermal monitoring
API Metrics: Request performance and endpoint health tracking
Production Monitor: System health scoring and alerting

3.5.5. Utilities

Model Inspector: Dynamic model capability detection
Hardware Optimizer: System optimization analysis and recommendations
Cost Calculator: Deployment cost analysis and planning

4. Features

4.1. Core Features

4.1.1. Multi-API Compatibility

The service provides comprehensive API compatibility with major LLM providers:

Figure 4. API Compatibility Architecture

Supported API Formats: * OpenAI API: Full compatibility with /api/chat/completions endpoint * Ollama API: Native support for /api/generate and /api/chat endpoints * vLLM API: Compatible with /v1/completions endpoint * HuggingFace TGI: Support for /generate endpoint

Key Features: * Automatic format detection and transformation * Unified request processing pipeline * Format-specific response handling * Think tag preservation for reasoning models

4.1.2. Dynamic Model Management

Figure 5. Model Management System

Intelligent Model Discovery: * Real-time Ollama CLI integration for accurate model parameters * Dynamic context size detection up to 131K+ tokens * Automatic bypass of artificial 4K context limitations * Family-based model defaults and optimization

Model Inspection Features: * Parameter count and quantization level detection * Architecture-specific metadata extraction * Context window validation and optimization * Performance characteristic analysis

4.1.3. GPU Management & Optimization

Figure 6. GPU Management Architecture

Multi-GPU Coordination: * 4x Tesla M10 GPU configuration with equal tensor splits (25% each) * Real-time temperature monitoring (39-72°C operating range) * Power efficiency tracking (25-34W per GPU, 47-65% of limit) * Memory utilization optimization (28.5% current usage with room for scaling)

Performance Monitoring: * Live GPU utilization tracking (38.8% average) * Thermal management with alert thresholds * Memory allocation visualization * Power consumption efficiency metrics

4.2. Advanced Features

4.2.1. Think Tag Preservation

Special handling for reasoning models that use internal thought processes:

Figure 7. Think Tag Processing Flow

Processing Logic: * Ollama Format: Preserves <think>…</think> tags in responses (matches official behavior) * OpenAI Format: Strips think tags for clean responses, stores content for analysis * Model Support: phi4-reasoning, deepseek-r1, and other reasoning models * Transparency: Optional think content exposure for debugging and analysis

4.2.2. Real-time Monitoring Dashboard

Figure 8. Web Dashboard Features

Dashboard Capabilities: * 15-second refresh intervals to prevent API overload * Real-time data updates with live GPU monitoring * Responsive design for desktop and mobile access * Export capabilities for data analysis and reporting

Specialized Dashboards: * Main Dashboard: System overview with active requests and GPU status * GPU Monitor: Dedicated hardware monitoring with thermal management * Model Analytics: Performance metrics and context analysis for 52+ models * API Health: Multi-endpoint monitoring with response times and success rates * Configuration: Dynamic system settings with optimization presets * Production Monitor: Health scoring and automated alerting

4.2.3. Hardware Optimization System

Figure 9. Hardware Optimization Engine

Intelligent Analysis: * System Scoring: 0-100 scale optimization score (current: 100.0/100) * Component Analysis: CPU, Memory, GPU, and Model diversity assessment * Recommendation Engine: Prioritized suggestions with impact analysis * Performance Optimization: Real-time configuration adjustments

Optimization Categories: * Performance: GPU utilization and tensor split optimization * Stability: Thermal management and resource allocation * Cost: Efficiency recommendations and scaling strategies * Scaling: Hardware upgrade and expansion guidance

4.3. Dashboard Screenshots

4.3.1. Main Dashboard

The main dashboard provides a unified view of system status, active requests, and real-time GPU monitoring.

4.3.2. GPU Monitor Dashboard

Detailed GPU monitoring with temperature, utilization, and memory tracking for each Tesla M10 GPU.

4.3.3. Configuration Panel

Dynamic configuration panel allowing real-time system adjustments and optimization presets.

4.3.4. Optimization Dashboard

Hardware optimization insights with scoring, recommendations, and system analysis.

4.4. Production Features

4.4.1. Model Analytics

Performance Benchmarking: Comprehensive model comparison and analysis
Context Detection: Automatic identification of true context capabilities
Usage Patterns: Statistical analysis of model utilization
Optimization Recommendations: Model-specific performance tuning

4.4.2. API Metrics Tracking

Real-time Performance: Actual endpoint response times and success rates
Error Analysis: Detailed failure tracking and categorization
Usage Statistics: Request volume and pattern analysis
Health Monitoring: Automated endpoint health assessment

4.4.3. Configuration Management

Dynamic Updates: Real-time configuration changes without restart
Preset Profiles: High Performance, Balanced, and Conservative modes
Validation: Configuration impact analysis and validation
Backup/Restore: Configuration versioning and rollback capabilities

4.4.4. Cost Analysis

Deployment Planning: TCO analysis for hardware configurations
ROI Calculator: Cost-benefit analysis for different setups
Efficiency Metrics: Performance per dollar optimization
Scaling Recommendations: Cost-effective expansion strategies

4.5. Integration Features

4.5.1. External Service Integration

Ollama CLI: Direct integration for model inspection and management
NVIDIA Drivers: Real-time GPU monitoring and management
SystemD: Production service lifecycle management
Prometheus: Metrics export for external monitoring systems

4.5.2. Development Tools

Debug Mode: Enhanced logging and troubleshooting capabilities
Testing Framework: Comprehensive API and system testing
Documentation: Auto-generated API documentation and examples
Development Server: Hot-reload development environment

5. API Reference

5.1. Overview

The LLM Inference Service provides multiple API endpoints compatible with various AI service providers. All endpoints support both streaming and non-streaming responses.

5.2. Authentication

Currently, the service does not require authentication. In production deployments, implement appropriate authentication mechanisms.

5.3. Base URL

http://localhost:11435

5.4. Chat Completion APIs

5.5. OpenAI Compatible

Create Chat Completion

POST /api/chat/completions
Content-Type: application/json

{
  "model": "phi4:latest",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 200,
  "stream": false
}

Response:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "phi4:latest",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I'm doing well, thank you! How can I help you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 15,
    "total_tokens": 35
  }
}

5.6. Ollama Compatible

Generate Completion

POST /api/generate
Content-Type: application/json

{
  "model": "llama3:70b",
  "prompt": "Why is the sky blue?",
  "stream": false,
  "options": {
    "temperature": 0.8,
    "num_predict": 100,
    "num_ctx": 8192
  }
}

Chat Completion

POST /api/chat
Content-Type: application/json

{
  "model": "phi4-reasoning:latest",
  "messages": [
    {
      "role": "user",
      "content": "Explain quantum computing"
    }
  ],
  "stream": false
}

5.7. vLLM Compatible

Create Completion

POST /v1/chat/completions
Content-Type: application/json

{
  "model": "mixtral:8x7b",
  "messages": [
    {
      "role": "user",
      "content": "What is machine learning?"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 150
}

5.8. HuggingFace TGI Compatible

Generate Text

POST /generate
Content-Type: application/json

{
  "inputs": "The future of AI is",
  "parameters": {
    "max_new_tokens": 50,
    "temperature": 0.9,
    "top_p": 0.95
  }
}

5.9. Model Management APIs

5.10. List Models

GET /api/models

Response:

{
  "models": [
    {
      "name": "phi4:latest",
      "id": "phi4:latest",
      "parameter_size": "14B",
      "quantization": "Q8_0",
      "context_size": 16384,
      "created": "2025-06-07T12:00:00Z"
    },
    {
      "name": "llama3:70b",
      "id": "llama3:70b",
      "parameter_size": "70B",
      "quantization": "Q4_K_M",
      "context_size": 8192,
      "created": "2025-06-07T12:00:00Z"
    }
  ]
}

5.11. Show Model Details

POST /api/show
Content-Type: application/json

{
  "name": "phi4:latest"
}

Response:

{
  "license": "Apache 2.0",
  "modelfile": "# Modelfile generated...",
  "parameters": "parameter_size 14B\nquantization_level Q8_0",
  "template": "{{- if .System }}{{ .System }}{{ end }}...",
  "details": {
    "format": "gguf",
    "family": "phi",
    "parameter_size": "14B",
    "quantization_level": "Q8_0"
  },
  "model_info": {
    "general.architecture": "phi",
    "phi.context_length": 16384,
    "phi.embedding_length": 4096
  }
}

5.12. Monitoring APIs

5.13. Health Check

GET /health

Response:

{
  "status": "healthy",
  "timestamp": "2025-06-07T20:30:00Z",
  "components": {
    "executor": "healthy",
    "models": "52 available",
    "active_requests": 3
  }
}

5.14. Dashboard Metrics

GET /api/metrics/dashboard

Response:

{
  "status": {
    "status": "healthy",
    "timestamp": "2025-06-07T20:30:00Z"
  },
  "requests": [
    {
      "request_id": "abc123",
      "status": "processing",
      "model": "phi4:latest",
      "progress": 45,
      "total": 100
    }
  ],
  "models": [...],
  "gpu_metrics": {
    "gpus": [...],
    "total_memory_used": 9120,
    "total_memory_available": 32768
  }
}

5.15. GPU Metrics

GET /api/metrics/gpu

Response:

{
  "timestamp": "2025-06-07T20:30:00Z",
  "gpus": [
    {
      "index": 0,
      "name": "Tesla M10",
      "temperature": 45,
      "utilization_percent": 35.5,
      "memory_used": 2280,
      "memory_total": 8192,
      "power_draw": 30,
      "power_limit": 225
    }
  ],
  "driver_version": "535.154.05",
  "cuda_version": "12.2"
}

5.16. API Endpoint Metrics

GET /api/metrics/apis

5.17. Hardware Optimization Insights

GET /api/metrics/optimization

5.18. Configuration APIs

5.19. Update Configuration

POST /api/dashboard/configure
Content-Type: application/json

{
  "tensor_split": "0.25,0.25,0.25,0.25",
  "gpu_layers": 999,
  "context_size": 131072,
  "batch_size": 512
}

5.20. Weight Distribution Management

GET /api/dashboard/weight-distribution
POST /api/dashboard/weight-distribution

{
  "preset": "balanced",
  "model_name": "llama3:70b"
}

5.21. Progress Tracking

5.22. Get Request Progress

GET /api/progress/{request_id}

5.23. Dismiss Request

POST /api/dismiss/{request_id}

5.24. Response Formats

5.25. Streaming Responses

When stream: true is set, responses are sent as Server-Sent Events (SSE):

data: {"choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"choices":[{"delta":{"content":" there"},"index":0}]}

data: [DONE]

5.26. Error Responses

Warning

All API errors return appropriate HTTP status codes and structured error messages for easy debugging.

{
  "error": {
    "message": "Model not found",
    "type": "model_not_found",
    "code": 404
  }
}

5.27. Rate Limiting

Currently no rate limiting is implemented. In production, implement appropriate rate limiting based on your requirements.

5.28. Best Practices

Note	Following these practices ensures optimal performance and reliability:

Model Selection: Use the /api/models endpoint to discover available models before making requests
Context Limits: Check model capabilities with /api/show to understand context window sizes
Streaming: Use streaming for better user experience with long responses
Error Handling: Implement proper error handling for all API calls
Monitoring: Use the dashboard metrics endpoints to monitor system health

6. Deployment

6.1. Overview

This guide covers deployment options for the LLM Inference Service in production environments.

6.2. System Requirements

6.3. Hardware Requirements

Component	Minimum	Recommended
CPU	8 cores, x86_64	32+ cores, Intel Xeon or AMD EPYC
Memory	32GB RAM	256GB+ ECC RAM
GPU	1x NVIDIA GPU (8GB VRAM)	4x NVIDIA GPUs (24GB+ VRAM each)
Storage	500GB SSD	2TB+ NVMe SSD
Network	1 Gbps	10 Gbps

6.4. Software Requirements

Ubuntu 20.04+ or RHEL 8+
Python 3.10+
NVIDIA Driver 525+
CUDA 11.8+
Docker 20.10+ (optional)
SystemD (for service deployment)

6.5. Deployment Architecture

Figure 10. Deployment Options Overview

The service supports multiple deployment patterns:

SystemD Service: Native installation with automatic lifecycle management
Container Deployment: Docker-based deployment with GPU passthrough
High Availability: Multi-instance setup with load balancing
Cloud-Native: Kubernetes deployment for scalable cloud environments

6.6. SystemD Service Deployment

6.7. Installation Steps

Clone the repository:

git clone https://github.com/your-org/llm-inference-service.git
cd llm-inference-service

Install dependencies:

# System dependencies
sudo apt-get update
sudo apt-get install -y python3-pip python3-venv nvidia-driver-535

# Python environment
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Build llama.cpp:

cd /opt/llm/models/ollama-custom-models
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_CUDA=ON
make -j$(nproc)

Configure the service:

# Copy and edit configuration
cp config/service_config.example.yaml config/service_config.yaml
vim config/service_config.yaml

Install SystemD service:

# For system-wide installation
sudo ./install-service.sh

# For user-level installation
./install-user-service.sh

Start the service:

# System service
sudo systemctl start llm-inference
sudo systemctl enable llm-inference

# User service
systemctl --user start llm-inference
systemctl --user enable llm-inference

6.8. Service Configuration

# /opt/llm/inference-service/config/service_config.yaml
models_dir: /opt/llm/models/ollama/models/blobs
manifests_dir: /opt/llm/models/ollama/models/manifests
llama_cpp_path: /opt/llm/models/ollama-custom-models/llama.cpp
default_context_size: 131072
tensor_split: '0.25,0.25,0.25,0.25'  # For 4 GPUs
gpu_layers: 999
threads: 32
batch_size: 512
request_timeout: 1800
log_level: INFO
enable_model_inspection: true
preserve_think_tags: true

6.9. Service Management

# Check status
systemctl --user status llm-inference

# View logs
journalctl --user -u llm-inference -f

# Restart service
systemctl --user restart llm-inference

# Stop service
systemctl --user stop llm-inference

6.10. Docker Deployment

6.11. Building the Image

# Dockerfile
FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04

# Install dependencies
RUN apt-get update && apt-get install -y \
    python3.10 \
    python3-pip \
    git \
    cmake \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy application
COPY . .

# Install Python dependencies
RUN pip3 install --no-cache-dir -r requirements.txt

# Build llama.cpp
RUN cd /opt && \
    git clone https://github.com/ggerganov/llama.cpp && \
    cd llama.cpp && \
    mkdir build && cd build && \
    cmake .. -DLLAMA_CUDA=ON && \
    make -j$(nproc)

# Expose port
EXPOSE 11435

# Run the service
CMD ["python3", "-m", "ollama_server.main", "--port", "11435"]

6.12. Docker Compose

version: '3.8'

services:
  llm-inference:
    build: .
    image: llm-inference:latest
    container_name: llm-inference
    ports:
      - "11435:11435"
    volumes:
      - /opt/llm/models:/opt/llm/models
      - ./config:/app/config
      - ./logs:/app/logs
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - CUDA_VISIBLE_DEVICES=0,1,2,3
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

6.13. Running with Docker

# Build the image
docker-compose build

# Start the service
docker-compose up -d

# View logs
docker-compose logs -f

# Stop the service
docker-compose down

6.14. High Availability Setup

Figure 11. High Availability Architecture

6.15. Security Considerations

6.16. Network Security

SSL/TLS Configuration:

server {
    listen 443 ssl http2;
    server_name llm.example.com;

    ssl_certificate /etc/ssl/certs/llm.crt;
    ssl_certificate_key /etc/ssl/private/llm.key;

    location / {
        proxy_pass http://localhost:11435;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

6.17. Authentication

Implement authentication middleware:

from functools import wraps
from flask import request, jsonify

def require_api_key(f):
    @wraps(f)
    def decorated_function(*args, **kwargs):
        api_key = request.headers.get('X-API-Key')
        if not api_key or not validate_api_key(api_key):
            return jsonify({'error': 'Invalid API key'}), 401
        return f(*args, **kwargs)
    return decorated_function

6.18. Monitoring Setup

Note	The service includes comprehensive built-in monitoring dashboards: Real-time GPU monitoring with thermal management API endpoint health tracking Model performance analytics Hardware optimization insights Production monitoring with alerting

Note	The service includes built-in web dashboards with all monitoring capabilities. External monitoring integration (Prometheus/Grafana) is optional for advanced use cases.

6.19. Backup and Recovery

6.20. Backup Strategy

#!/bin/bash
# backup.sh

BACKUP_DIR="/backup/llm-inference"
DATE=$(date +%Y%m%d_%H%M%S)

# Backup configuration
cp -r /opt/llm/inference-service/config $BACKUP_DIR/config_$DATE

# Backup logs
tar -czf $BACKUP_DIR/logs_$DATE.tar.gz /opt/llm/inference-service/logs

# Backup model metadata (not the models themselves)
cp -r /opt/llm/models/ollama/models/manifests $BACKUP_DIR/manifests_$DATE

echo "Backup completed: $BACKUP_DIR/*_$DATE"

6.21. Recovery Procedure

Restore configuration files
Restart the service
Verify model availability
Test API endpoints

6.22. Troubleshooting

6.23. Common Issues

Caution

Most issues can be resolved by checking logs and verifying configuration. Always check logs first!

Issue	Cause	Solution
Service won’t start	Missing dependencies	Check logs with `journalctl --user -u llm-inference -n 100`
GPU not detected	Driver issues	Verify with `nvidia-smi` and reinstall drivers if needed
High memory usage	Large context windows	Adjust `default_context_size` in configuration
Slow inference	CPU-only mode	Check `gpu_layers` setting and GPU availability
Model not found	Incorrect path	Verify `models_dir` in configuration

6.24. Debug Mode

Enable debug logging:

# Edit service config
log_level: DEBUG

# Or run manually
python -m ollama_server.main --debug --port 11435

6.25. Performance Tuning

6.26. GPU Optimization

# Optimal settings for 4x Tesla M10
tensor_split: '0.25,0.25,0.25,0.25'
gpu_layers: 999
batch_size: 512

6.27. Context Management

# Balance between capability and memory
default_context_size: 32768  # For limited VRAM
# default_context_size: 131072  # For ample VRAM

6.28. Request Handling

# Concurrent request tuning
request_timeout: 1800
max_concurrent_requests: 10  # Adjust based on GPU memory

6.29. Production Checklist

6.30. Pre-Deployment

❏ Hardware requirements verified
❏ Dependencies installed and tested
❏ Configuration file reviewed and customized
❏ SSL certificates configured (if applicable)
❏ Network security configured
❏ Backup strategy defined

6.31. Post-Deployment

❏ Service health check passing
❏ GPU monitoring active
❏ API endpoints responsive
❏ Dashboard accessible
❏ Logging configuration verified
❏ Monitoring alerts configured
❏ Performance benchmarks established

6.32. Ongoing Maintenance

❏ Regular log rotation
❏ Performance monitoring
❏ Security updates
❏ Configuration backups
❏ Capacity planning reviews == Monitoring

6.33. Overview

The LLM Inference Service provides comprehensive monitoring capabilities for production deployments.

6.34. Monitoring Architecture

Figure 12. Monitoring System Architecture

The monitoring system consists of:

LLM Inference Service: Core service with integrated metrics collection
Metrics Collection: Prometheus exporters and log aggregation
Visualization: Web dashboards and Grafana integration
Storage: Time-series database for metrics and log storage
Alerting: Automated alert management and notifications

6.35. Web Dashboard

6.36. Dashboard Overview

The service includes a comprehensive web dashboard accessible at http://localhost:11435/dashboard.

Figure 13. Dashboard Pages Architecture

Dashboard Features: * 15-second auto-refresh to prevent API overload * Real-time data updates with live GPU monitoring * Responsive design for desktop and mobile access * Export capabilities for data analysis and reporting

6.37. Key Metrics Tracked

Metric	Description	Update Frequency
GPU Temperature	Per-GPU temperature in Celsius	Real-time (1s)
GPU Utilization	Percentage of GPU compute usage	Real-time (1s)
Memory Usage	VRAM usage per GPU in MB	Real-time (1s)
Request Latency	End-to-end request processing time	Per request
Token Generation Rate	Tokens per second during inference	Per request
API Success Rate	Percentage of successful requests	1 minute window
System Health Score	0-100 composite health indicator	1 minute
Active Alerts	Current unresolved system alerts	Real-time

6.38. Production Monitoring

6.39. Health Score Calculation

Figure 14. Health Score Calculation Flow

Scoring Components: * CPU Score: 100 - CPU utilization percentage * Memory Score: 100 - Memory utilization percentage * GPU Score: 100 - (Temperature/Max × 100) * Error Score: 100 - (Error rate × 10) * Response Score: 100 - ((Response time - 5000) / 250)

Final Calculation: * Weighted average of all component scores * Penalty reduction for active alerts (20 points per alert) * Scale: 0-100 (>80 = Healthy, 60-80 = Warning, <60 = Critical)

6.40. Alert Configuration

Configure alerts in the Production Monitor dashboard:

# Alert thresholds
alerts:
  cpu_usage:
    warning: 80    # percentage
    critical: 95
    duration: 300  # seconds

  memory_usage:
    warning: 85
    critical: 95
    duration: 300

  gpu_temperature:
    warning: 80    # Celsius
    critical: 85
    duration: 180

  error_rate:
    warning: 5     # percentage
    critical: 15
    duration: 120

  response_time:
    warning: 10000 # milliseconds
    critical: 30000
    duration: 180

6.41. Alert Types

Figure 15. Alert Processing Flow

Alert Lifecycle: 1. Metric Collection: Continuous monitoring of system metrics 2. Threshold Evaluation: Check against warning and critical thresholds 3. Duration Timer: Prevent false positives with sustained threshold violations 4. Alert Creation: Generate warning or critical alerts based on severity 5. Notification: Send alerts via dashboard, logs, email, or webhooks 6. Resolution: Automatic resolution when metrics return to normal

7. GPU Monitoring

7.1. Metrics Collected

{
  "timestamp": "2025-06-07T20:30:00Z",
  "gpus": [
    {
      "index": 0,
      "name": "Tesla M10",
      "temperature": 45,
      "utilization_percent": 35.5,
      "memory_used": 2280,
      "memory_total": 8192,
      "memory_percent": 27.8,
      "power_draw": 30,
      "power_limit": 225,
      "fan_speed": 40
    }
  ],
  "total_memory_used": 9120,
  "total_memory_available": 32768,
  "average_temperature": 48.5,
  "average_utilization": 38.8,
  "thermal_status": "good"
}

7.2. Thermal Management

Temperature Range	Status	Action
< 70°C	Good	Normal operation
70-80°C	Warning	Monitor closely, check cooling
80-85°C	Critical	Reduce workload, check fans
> 85°C	Emergency	Throttle or shutdown

7.3. API Monitoring

7.4. Endpoint Metrics

Track performance for each API endpoint:

{
  "endpoints": [
    {
      "endpoint": "OpenAI API",
      "path": "/api/chat/completions",
      "total_requests": 1527,
      "successful_requests": 1520,
      "failed_requests": 7,
      "average_response_time": 245.3,
      "success_rate": 99.5,
      "last_request_time": "2025-06-07T20:29:45Z",
      "status": "healthy"
    }
  ],
  "timestamp": "2025-06-07T20:30:00Z",
  "overallHealth": "healthy"
}

7.5. Performance Tracking

Figure 16. Performance Tracking Flow

Tracking Process: 1. Request Initiation: Client sends request to API gateway 2. Timer Start: Metrics collector starts response time measurement 3. Request Processing: API processes request through inference pipeline 4. Metrics Calculation: Calculate response time, success/failure, token count 5. Dashboard Update: Real-time dashboard updates with latest metrics

7.6. Log Management

7.7. Log Structure

{
  "timestamp": "2025-06-07T20:30:00.123Z",
  "level": "INFO",
  "logger": "ollama_server.api.handlers",
  "request_id": "7b4fa8e5-e2a9-4410-b43c-d071c8323fe1",
  "message": "Handling non-streaming request",
  "extra": {
    "api": "openai",
    "model": "phi4:latest",
    "endpoint": "/api/chat/completions",
    "remote_addr": "192.168.1.100"
  }
}

7.8. Log Levels

Level	Usage	Example
ERROR	System errors, failures	Model loading failed
WARNING	Performance issues, alerts	High GPU temperature
INFO	Normal operations	Request completed
DEBUG	Detailed troubleshooting	Token generation details

7.9. Log Rotation

Configure log rotation in /etc/logrotate.d/llm-inference:

/opt/llm/inference-service/logs/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 llm llm
    sharedscripts
    postrotate
        systemctl reload llm-inference
    endscript
}

7.10. Performance Optimization

7.11. Monitoring-Based Optimization

Figure 17. Optimization Decision Flow

Optimization Logic: * Low GPU Utilization (<50%): Increase batch size for better throughput * High Memory Usage (>90%): Reduce context size to prevent OOM errors * High Response Time: Increase GPU layers or add more GPUs * Optimal Performance: Maintain current configuration

7.12. Key Performance Indicators

Throughput: Tokens per second
Latency: 95th percentile response time
Efficiency: GPU utilization percentage
Reliability: Success rate percentage
Scalability: Concurrent request capacity

7.13. Alerting Integration

7.14. Webhook Configuration

# config/alerting.yaml
alerting:
  webhooks:
    - url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
      events: ["critical", "warning"]

    - url: "https://api.pagerduty.com/incidents"
      events: ["critical"]
      headers:
        Authorization: "Token token=YOUR_TOKEN"

7.15. Email Alerts

email:
  smtp_server: "smtp.gmail.com"
  smtp_port: 587
  use_tls: true
  username: "alerts@example.com"
  recipients:
    - "ops-team@example.com"
    - "on-call@example.com"

7.16. Grafana Integration

7.17. Prometheus Metrics Export

The service exposes Prometheus-compatible metrics at /metrics:

# HELP llm_request_duration_seconds Request duration
# TYPE llm_request_duration_seconds histogram
llm_request_duration_seconds_bucket{api="openai",model="phi4",le="0.1"} 245
llm_request_duration_seconds_bucket{api="openai",model="phi4",le="0.5"} 1203

# HELP llm_gpu_temperature_celsius GPU temperature
# TYPE llm_gpu_temperature_celsius gauge
llm_gpu_temperature_celsius{gpu="0",name="Tesla M10"} 45

# HELP llm_active_requests Number of active requests
# TYPE llm_active_requests gauge
llm_active_requests 3

7.18. Sample Grafana Dashboard

Import monitoring/grafana-dashboard.json for pre-configured panels:

Request rate and latency
GPU utilization heatmap
Model performance comparison
System resource usage
Alert history

7.19. Troubleshooting with Monitoring

7.20. Performance Issues

Check GPU utilization in dashboard
Review response time trends
Analyze error logs
Verify resource allocation

7.21. Common Patterns

Pattern	Likely Cause	Investigation
Increasing latency	Memory pressure, thermal throttling	Check GPU temperature and VRAM usage
Error rate spikes	Model issues, OOM errors	Review error logs and GPU memory
Low GPU utilization	CPU bottleneck, small batch size	Increase batch size, check CPU usage
Intermittent failures	Resource contention, driver issues	Check system logs and dmesg

7.22. Monitoring Best Practices

7.23. Dashboard Usage

Monitor main dashboard for system overview
Use specialized dashboards for detailed analysis
Set up automated alerts for critical metrics
Export data for trend analysis and capacity planning

7.24. Alert Configuration

Set appropriate warning and critical thresholds
Use duration-based triggering to prevent false positives
Configure multiple notification channels for redundancy
Regular review and tuning of alert thresholds

7.25. Performance Monitoring

Establish baseline performance metrics
Monitor trends for capacity planning
Use optimization insights for system tuning
Regular performance reviews and adjustments

7.26. Log Analysis

Configure appropriate log levels for environment
Implement log rotation to manage disk space
Use structured logging for better analysis
Regular log review for security and performance insights

8. Quick Start

8.1. Installation

Important

The service runs on port 11435 by default. Ensure this port is available before installation.

# Install user-level SystemD service
./install-user-service.sh

# Start the service (runs on port 11435)
systemctl --user start llm-inference

# Access web dashboard
open http://localhost:11435/dashboard

8.2. Test API

Tip	Use `tinyllama:1.1b-chat` for quick testing - it’s a small, fast model ideal for verifying your installation.

# List models
curl http://localhost:11435/api/models

# Test with fast model
curl -X POST http://localhost:11435/api/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "tinyllama:1.1b-chat", "messages": [{"role": "user", "content": "Hi"}]}'

8.3. Production Status (v1.0)

Component	Status	Details
Multi-API Support	✅ v1.0 Production Ready	OpenAI, Ollama, vLLM, HuggingFace TGI with auto model selection
GPU Monitoring	✅ v1.0 Production Ready	4x Tesla M10 with thermal management and power tracking
Dashboard System	✅ v1.0 Production Ready	6 specialized dashboards, 15s refresh, real-time metrics
Model Management	✅ v1.0 Production Ready	52+ models, up to 131K context detection, think tag support
Hardware Optimization	✅ v1.0 Production Ready	100.0/100 system score, intelligent recommendations, dynamic config
API Health Monitoring	✅ v1.0 Production Ready	Real-time endpoint tracking, response time analysis, error handling

8.4. Dashboard Pages Overview

Dashboard Page	Description	Access Path
Main Dashboard	Real-time system overview with GPU monitoring, active requests, model status, and service health	`/dashboard`
GPU Monitor	Dedicated GPU monitoring with temperature tracking, memory utilization, power consumption, and thermal status for 4x Tesla M10 GPUs	`/dashboard/gpu`
Model Analytics	Advanced model performance analysis, context length detection, parameter insights, and usage statistics for 52+ models	`/dashboard/models`
API Health Monitor	Live endpoint monitoring with response times, success rates, and usage metrics for OpenAI, Ollama, vLLM, and HuggingFace APIs	`/dashboard/apis`
Configuration Panel	Dynamic system configuration with GPU tensor splits, performance tuning, context sizes, and preset configurations	`/dashboard/config`
Hardware Optimization	Intelligent hardware analysis with optimization recommendations, system scoring, and performance insights	`/dashboard/optimization`

8.5. Dashboard Features

Figure 18. LLM Inference Service - Main Dashboard

Figure 19. GPU Monitor Dashboard

Figure 20. Configuration Panel

Figure 21. Hardware Optimization Dashboard

Key Features: * Real-time GPU monitoring (4x Tesla M10) * 15-second dashboard refresh intervals * Dynamic configuration management * Hardware optimization scoring (100.0/100 current score) * API metrics tracking with actual endpoint data * 52+ model management with context detection up to 131K tokens

9. License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

9.1. Third-Party Licenses

This project builds upon and includes code from:

llama.cpp: MIT License - High-performance LLM inference engine
Flask: BSD License - Web framework
Ollama: MIT License - Model format and API design inspiration

See the NOTICE file for complete license information and attributions.

10. Support

For support and questions:

Issues: Create an issue in the project repository
Documentation: Check this README and inline code documentation
Community: Join the discussion in project forums

11. Acknowledgments

This project builds upon the excellent work of:

llama.cpp: High-performance LLM inference engine
Ollama: Model format and API design inspiration
OpenAI: API compatibility standards
Flask: Web framework foundation

Name		Name	Last commit message	Last commit date
Latest commit History 48 Commits
.github		.github
app		app
diagrams		diagrams
docker		docker
docs		docs
images		images
ollama_server		ollama_server
scripts		scripts
.gitignore		.gitignore
.npmrc		.npmrc
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
NOTICE		NOTICE
README.adoc		README.adoc
diagram-config.json		diagram-config.json
generate-diagrams.sh		generate-diagrams.sh
generate-images.sh		generate-images.sh
generate-updated-diagrams.sh		generate-updated-diagrams.sh
install-service.sh		install-service.sh
install-user-service.sh		install-user-service.sh
llm-inference.service		llm-inference.service
package.json		package.json
puppeteer-config.json		puppeteer-config.json
requirements.txt		requirements.txt
service-status-user.sh		service-status-user.sh
service-status.sh		service-status.sh
test_context_fix.py		test_context_fix.py
test_refactored_server.py		test_refactored_server.py
uninstall-service.sh		uninstall-service.sh

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

LLM Inference Service v1.0

1. Overview

1.1. Why This Project Exists

1.2. What This Solution Provides

1.3. How It Works

2. v1.0 Release Notes

2.1. What’s New in v1.0

2.1.1. Major Features

2.1.2. Key Improvements

2.1.3. Technical Achievements

2.2. Deployment Status

2.3. Getting Started with v1.0

2.4. Migration from Beta

3. Architecture

3.1. Core Components

3.2. Component Interactions

3.3. Deployment Architecture

3.4. Technical Design Patterns

3.4.1. Adapter Pattern

3.4.2. Observer Pattern

3.4.3. Strategy Pattern

3.4.4. Factory Pattern

3.5. Component Details

3.5.1. API Gateway Layer

3.5.2. Request Processing

3.5.3. Inference Engine

3.5.4. Monitoring & Analytics

3.5.5. Utilities

4. Features

4.1. Core Features

4.1.1. Multi-API Compatibility

4.1.2. Dynamic Model Management

4.1.3. GPU Management & Optimization

4.2. Advanced Features

4.2.1. Think Tag Preservation

4.2.2. Real-time Monitoring Dashboard

4.2.3. Hardware Optimization System

4.3. Dashboard Screenshots

4.3.1. Main Dashboard

4.3.2. GPU Monitor Dashboard

4.3.3. Configuration Panel

4.3.4. Optimization Dashboard

4.4. Production Features

4.4.1. Model Analytics

4.4.2. API Metrics Tracking

4.4.3. Configuration Management

4.4.4. Cost Analysis

4.5. Integration Features

4.5.1. External Service Integration

4.5.2. Development Tools

5. API Reference

5.1. Overview

5.2. Authentication

5.3. Base URL

5.4. Chat Completion APIs

5.5. OpenAI Compatible

Create Chat Completion

5.6. Ollama Compatible

Generate Completion

Chat Completion

5.7. vLLM Compatible

Create Completion

5.8. HuggingFace TGI Compatible

Generate Text

5.9. Model Management APIs

5.10. List Models

5.11. Show Model Details

5.12. Monitoring APIs

5.13. Health Check

5.14. Dashboard Metrics

5.15. GPU Metrics

5.16. API Endpoint Metrics

5.17. Hardware Optimization Insights

5.18. Configuration APIs

5.19. Update Configuration