Skip to content

v1.0.2

Choose a tag to compare

View all tags
@Josh-XT Josh-XT released this 26 Nov 14:44
· 504 commits to main since this release
v1.0.2
e51f597

ezLocalai v1.0.2

Summary

This release represents a complete modernization of ezlocalai's AI stack, making it truly "ez" - simpler configuration, smarter defaults, and better performance. The result is a cleaner codebase with fewer dependencies, automatic optimization, and minimal required configuration.

Philosophy: VRAM is auto-detected, context is dynamic, GPU layers are auto-calibrated. Just set your models and go.

Key Changes

1. Dynamic Context Sizing

No more guessing context sizes! The system automatically sizes context to fit your prompt:

  • Estimates prompt tokens (chars / 4)
  • Rounds up to nearest 32k boundary (32k, 64k, 96k, 128k)
  • Recalibrates GPU layers if context increases
  • Result: Much faster inference on short prompts (128k→32k = 63% speedup)

2. Auto VRAM Detection

VRAM budget is now detected automatically at startup:

  • Queries torch.cuda.get_device_properties()
  • Rounds down to nearest GB for safety margin
  • No more manual VRAM_BUDGET configuration

3. Simplified Environment Variables

Removed env vars (now automatic):

Variable Reason
GPU_LAYERS Auto-calibrated based on VRAM
VRAM_BUDGET Auto-detected from GPU
LLM_MAX_TOKENS Dynamic based on prompt size
IMG_ENABLED Detected from IMG_MODEL being set
EMBEDDING_ENABLED Always available
IMG_DEVICE Auto-detects CUDA availability

Renamed:

Old New Reason
SD_MODEL IMG_MODEL Clearer naming

Simplified DEFAULT_MODEL format:

# Old: model@tokens,model@tokens
DEFAULT_MODEL=model1@64000,model2@8192

# New: just list models (context is dynamic)
DEFAULT_MODEL=model1,model2

4. Vision Model Fallback

When a non-vision model receives an image request:

  1. System detects images in request
  2. Finds a vision-capable model from available models
  3. Uses vision model to describe images
  4. Prepends description to prompt
  5. Processes with requested model

This allows coding models (non-vision) to receive image context!

5. LLM Engine: llama-cpp-python → xllamacpp

Replaced llama-cpp-python with xllamacpp:

  • Unified LLM + Vision via multimodal projector (mmproj)
  • Native estimate_gpu_layers() function for fast calibration
  • Cleaner API through xllamacpp.Server

6. Multi-Model Hot-Swap

Support for multiple LLMs with automatic hot-swap:

DEFAULT_MODEL=unsloth/Qwen3-VL-4B-Instruct-GGUF,unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF
  • First model loads at startup with baseline 32k context
  • Models swap on demand based on request
  • Context-aware: recalibrates GPU layers when context size changes

7. Pre-Calibration at Startup

All models in DEFAULT_MODEL are pre-calibrated at startup:

  • Uses xllamacpp's native estimate_gpu_layers()
  • Calibrates at 32k baseline context
  • Caches results for instant swaps
  • Recalibrates on-demand for larger contexts

8. Text-to-Speech: XTTS → Chatterbox TTS

Replaced XTTS with Chatterbox TTS:

  • Modern Llama 3.2 backbone (0.5B params)
  • Better voice cloning quality
  • Preloaded at startup to warm cache (~38s → ~5s on reload)
  • Lazy-loaded on demand, unloaded after to free VRAM

9. Image Generation: SDXL-Turbo → SDXL-Lightning

Migrated to ByteDance/SDXL-Lightning:

  • Better image quality at low step counts
  • GPU: 2-step generation (blazing fast)
  • CPU: 4-step generation (stable)
  • Auto-detects CUDA for device selection

10. Embeddings: ONNX → BGE-M3

Replaced ONNX embedding with native BAAI/bge-m3:

  • Removed ONNX/Optimum dependencies
  • 1024-dimensional embeddings
  • Native PyTorch GPU acceleration

11. Pip-Installable CLI

New lightweight CLI for easy installation and management:

pip install ezlocalai
ezlocalai start

Features:

  • Auto-detects GPU (NVIDIA) or falls back to CPU mode
  • Auto-installs prerequisites on Linux (Docker, NVIDIA Container Toolkit)
  • Simple commands: start, stop, restart, status, logs
  • Configurable: --model, --uri, --api-key, --ngrok, --whisper, --img-model
  • Minimal dependencies: Only click and requests (no heavy ML libraries)
# Examples
ezlocalai start --model unsloth/gemma-3-4b-it-GGUF
ezlocalai start --api-key my-secret-key --ngrok <token>
ezlocalai logs -f

Performance Benchmarks

Test System

  • CPU: 12th Gen Intel Core i9-12900KS (24 cores)
  • GPU: NVIDIA GeForce RTX 4090 (24GB VRAM)
  • Models Tested:
    • unsloth/Qwen3-VL-4B-Instruct-GGUF - 4B vision model (Q4_K_XL quantization)
    • unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF - 30B MoE coding model (Q4_K_XL quantization)

LLM Inference Performance (RTX 4090)

Model Size Tokens Time Speed
Qwen3-VL-4B 4B 100 0.47s 213 tok/s
Qwen3-VL-4B 4B 134 0.64s 209 tok/s
Qwen3-Coder-30B 30B (MoE) 100 1.53s 65 tok/s
Qwen3-Coder-30B 30B (MoE) 300 4.41s 68 tok/s

Key Observations

  • 4B Vision Model: ~210 tok/s average - excellent for interactive use
  • 30B Coding Model: ~65-68 tok/s - great for code generation despite size
  • Hot-swap time: ~1s between models (pre-calibrated)
  • Dynamic context: 32k baseline, scales up as needed

GPU Layer Calibration

Auto-calibrated at startup for 32k context:

Model GPU Layers VRAM Usage
Qwen3-VL-4B 37 layers ~12GB
Qwen3-Coder-30B 45 layers ~24GB

Minimal Configuration

.env file (that's it!):

# ezlocalai Configuration - keeping it "ez"
# VRAM is auto-detected, context is dynamic, GPU layers are auto-calibrated

MAIN_GPU=0
DEFAULT_MODEL=unsloth/Qwen3-VL-4B-Instruct-GGUF,unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF

# Image generation (set to enable, leave empty to disable)
IMG_MODEL=

# Speech-to-text model
WHISPER_MODEL=base

# Queue settings
MAX_CONCURRENT_REQUESTS=2

API Compatibility

All API endpoints remain unchanged. This is a drop-in upgrade.

New CLI

A lightweight pip-installable CLI for managing ezlocalai Docker containers:

pip install ezlocalai
ezlocalai start

The CLI:

  • Auto-detects GPU and uses CUDA or CPU mode appropriately
  • Auto-installs prerequisites (Docker, NVIDIA Container Toolkit) on Linux
  • Builds CUDA image locally from source (too large for DockerHub)
  • Persists data in ~/.ezlocalai/data/ (models, outputs, voices)
  • No prompts - designed to be as "ez" as possible

Commands:

  • ezlocalai start [--model X] - Start with optional model override
  • ezlocalai stop - Stop the container
  • ezlocalai restart - Restart with same config
  • ezlocalai status - Show running status and configuration
  • ezlocalai logs [-f] - View container logs
  • ezlocalai update - Pull latest CPU image / rebuild CUDA image
Assets 2
Loading