Skip to content

v1.1.9

Choose a tag to compare

View all tags
@JamesXYZ321 JamesXYZ321 released this 10 Jul 10:30
· 649 commits to main since this release
v1.1.9
e52fe28

📦 Release Summary: Multi-Model Serving & Enhanced CLI

🚀 New Features

✅ Multi-Model Serving System

  • Introduced eai model serve — launch all downloaded models simultaneously with a single command.
  • New --main-hash flag to designate the primary model (automatically selected at random if unspecified).
  • Zero-config discovery of models from the llms-storage directory — no manual setup required.
  • Native support for mixed task types in a unified server instance:
    • 🧠 Chat
    • 🧬 Embedding
    • 🖼️ Image Generation
      Coming soon: 🎞️ Video Generation, 🎙️ Audio Recognition, and more!

🤖 New Models

  • Devstral-Small by Mistral AI is now supported!
    A compact, high-performance coding model designed for developer productivity — combining speed and efficiency with low resource usage.
    Preserved on Lighthouse by EternalAI with content hash:
    bafkreicr66zguzdldiqal4jdyrcd6y26lfaiy4glkah7n3hdve2ailptue
    👉 View in CryptoModels

🛠️ Enhanced Model Management

  • Smarter validation with improved error messaging.
  • Automatic fallback to the main model when a requested model is missing or invalid.
  • Dynamic per-request model switching using the model field (by content hash).

Switching Logic Flow:

Incoming Request
    ↓
Check Model Field
    ↓
Model Field Present?
    ├─ NO → Use Main Model (immediate)
    └─ YES → Is Requested Model Valid?
        ├─ NO → Use Main Model (fallback)
        └─ YES → Is Requested Model Active?
            ├─ YES → Process Request Immediately
            └─ NO → Check for Active Streams
                ├─ Streams Active → Wait for Completion
                └─ No Streams → Switch Model → Process Request

📖 Documentation Improvements

📚 Multi-Model User Guide

  • Comprehensive setup and usage guide for eai model serve.
  • Endpoint-specific examples for:
    • POST /v1/chat/completions
    • POST /v1/embeddings
    • POST /v1/images/generations
  • In-depth explanation of model-switching mechanics and fallback flow.
  • Tips for performance tuning and deployment scaling.

📑 API Clarity

  • Extended /v1/models documentation (thanks to @ihubanov in PR #7).
  • Transparent mapping from discovered models to routing logic.
  • Switched from folder-based to hash-based model identification.
  • Included flowcharts and diagrams to visualize request handling and model selection.

🔧 Technical Enhancements

  • Automatic fallback model selection when --main-hash is omitted.
  • Refined CLI UX with improved help menus and clearer command-line options.
  • More robust discovery logic and validation for downloaded models.
  • Cleaner error reporting for missing or malformed model hashes.

💡 Key Benefits

  • One Command to Rule Them All — serve multiple models effortlessly.
  • Hot Model Switching — swap models on-the-fly with no restart.
  • Built for Devs — intuitive CLI, detailed docs, and great DX.

This release marks a major leap toward flexible multi-model deployments.
Dive in and explore the new capabilities today!

Contributors

ihubanov
Assets 2
Loading