Skip to content

v0.2.5rc2

Pre-release
Pre-release

Choose a tag to compare

View all tags
@Chen-zexi Chen-zexi released this 24 Aug 22:10
· 8 commits to main since this release
v0.2.5rc2
65ed145

Multi-Model Proxy Server (Experimental)

The Multi-Model Proxy is a new experimental feature that enables serving multiple LLMs through a single unified API endpoint. This feature is currently under active development and available for testing.

What It Does:

  • Single Endpoint - All your models accessible through one API
  • Live Management - Add or remove models without stopping the service
  • Dynamic GPU Management - Efficient GPU resource distribution through vLLM's sleep/wake functionality
  • Interactive Setup - User-friendly wizard guides you through configuration

You can install the pre-release version with:

pip install --pre --upgrade vllm-cli
Assets 2
Loading