Skip to content

ReadyLLM v0.1.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@htobty htobty released this 16 Sep 05:55
· 11 commits to main since this release
v0.1.0
8587a3d

First public release of ReadyLLM — a web-based assistant for deploying, monitoring and tuning local LLMs, with ComfyUI image / video generation on top.

Out of the box, ollama often leaves most of a model on the CPU. ReadyLLM finds the parameters that actually fit your hardware and keeps watching it live. Measured on an RTX 4090, same model, same quantization, same context length: peak throughput +37%, average +33%, 100K context +50%.

Highlights

  • Unified target management — local machine or remote LAN hosts over SSH (key or password), OS auto-detected, inference engine and model directory configured per target.
  • Hardware detection & model recommendation — probes GPU / VRAM / CPU / RAM on the target and recommends models that actually fit.
  • Model store — curated built-in GGUF catalog plus dynamic HuggingFace popular-model listing; three download sources (hf-mirror, ModelScope, HuggingFace), background downloads with progress and post-download integrity checks.
  • One-click deployment — start / stop llama.cpp and vLLM services with editable runtime parameters and port; missing engines can be installed from within the tool (auto download / compile / pip, then back-fill the path).
  • Real-time monitoring — GPU utilization, VRAM, throughput and latency curves pushed live over WebSocket, plus per-day token usage stats (input / output) persisted across backend restarts.
  • Smart tuning
    • Auto tuning: two-phase search (coarse over high-impact discrete params, fine over continuous ones) with VRAM feasibility pre-checks and real benchmarking, targeting latency / throughput / prefill.
    • AI Agent tuning: plug in your own LLM API and let it reason out parameters from hardware / model / scenario, feeding benchmark results back for multi-round iteration.
  • Image & video generation (ComfyUI) — text-to-video, storyboard generation, long-video per-shot I2V serial generation and stitching, single-image and finished-video upscaling.
  • i18n — English by default, switchable to Simplified Chinese.

Requirements

  • Python 3.9+
  • Node.js 18+
  • Target machine has (or can install via this tool) llama.cpp / vLLM / ComfyUI

The control side runs on macOS / Linux / Windows. Note that vLLM does not run natively on Windows — Windows targets must go through WSL2 (the tool shows a guidance prompt).

Quick start

# backend
cd backend && pip install -r requirements.txt
uvicorn app.main:app --host 127.0.0.1 --port 8000

# frontend
cd frontend && npm install && npm run dev

Then open http://localhost:3000, add a target machine, pick a model, hit Tune.

Known limitations in this release

  • ComfyUI on the target listens on its own loopback address by default; when the control side cannot reach its port directly, finished-clip preview is proxied through the backend.
  • Model sizes are approximate — used only for display and VRAM filtering; the source repository is authoritative.
  • Pre-release: the tuning and benchmark paths have been validated on a single NVIDIA consumer GPU (RTX 4090). Other hardware, multi-GPU and AMD are untested.

Full setup, architecture and configuration: README · 简体中文

7 commits since the initial commit (f7d48b2 .. 8587a3d). Source snapshot: tar.gz · zip

Assets 2
Loading