Skip to content

v0.5.0

Latest

Choose a tag to compare

@Freim32 Freim32 released this 02 Oct 10:14
a374b26

Highlights

One gateway in front of every model. vllmops proxy start generates a LiteLLM config from your catalog and serves every running model behind a single OpenAI-compatible URL. The gateway follows the catalog: every start, stop and restart, from the CLI or the TUI, regenerates the config and respawns the gateway when it changed.

  • vllmops proxy start | stop | restart | status | config | logs
  • the whole gateway shape lives in the new proxy section of .vllmops/config.yaml (host, port, profile, which models to expose, workers)
  • .vllmops/litellm.yaml is merged into the generated config, so you can add routes and settings LiteLLM supports; proxy start lists those routes as (overlay)
  • LITELLM_MASTER_KEY in your shell or .env turns on authentication; the generated file only holds the os.environ/ reference, never the key
  • the TUI header shows the gateway state: proxy ● :4000, ▴ when the config is out of date, ◌ when down

Install LiteLLM as its own tool, outside the project, so its pinned dependencies cannot move vLLM's:

uv tool install 'litellm[proxy]'

Reliability

  • a proxy restart whose new config cannot start (empty profile, port taken by a model) fails and leaves the running gateway in place
  • an automatic refresh reports success only once the new gateway answers
  • the gateway keeps one log across respawns, each start marked, rotated past 50 MB
  • vllmops status lists a broken model file as an invalid row instead of failing; start on it says why it is invalid instead of "unknown model"
  • an invalid .vllmops/config.yaml is reported in one line instead of a traceback
  • CLI output prints brackets verbatim (log lines, vllmops command, hints like litellm[proxy]), keeps copyable commands on one line, and sends diagnostics to stderr so vllmops proxy config > file.yaml is valid YAML
  • the TUI counts a model that belongs to several profiles once
  • vllmops doctor checks the LiteLLM executable and flags models declared on the gateway port

Under the hood

  • type checking with mypy strict and pyright on sources and tests
  • CI on Python 3.10 through 3.14

Upgrade

uv tool upgrade vllmops     # or: pipx upgrade vllmops

Existing .vllmops/config.yaml files keep working: the proxy section is optional and has defaults.