Repository navigation
Highlights
One gateway in front of every model. vllmops proxy start generates a LiteLLM config from your catalog and serves every running model behind a single OpenAI-compatible URL. The gateway follows the catalog: every start, stop and restart, from the CLI or the TUI, regenerates the config and respawns the gateway when it changed.
vllmops proxy start | stop | restart | status | config | logs- the whole gateway shape lives in the new
proxysection of.vllmops/config.yaml(host, port, profile, which models to expose, workers) .vllmops/litellm.yamlis merged into the generated config, so you can add routes and settings LiteLLM supports;proxy startlists those routes as(overlay)LITELLM_MASTER_KEYin your shell or.envturns on authentication; the generated file only holds theos.environ/reference, never the key- the TUI header shows the gateway state:
proxy ● :4000,▴when the config is out of date,◌when down
Install LiteLLM as its own tool, outside the project, so its pinned dependencies cannot move vLLM's:
uv tool install 'litellm[proxy]'
Reliability
- a
proxy restartwhose new config cannot start (empty profile, port taken by a model) fails and leaves the running gateway in place - an automatic refresh reports success only once the new gateway answers
- the gateway keeps one log across respawns, each start marked, rotated past 50 MB
vllmops statuslists a broken model file as aninvalidrow instead of failing;starton it says why it is invalid instead of "unknown model"- an invalid
.vllmops/config.yamlis reported in one line instead of a traceback - CLI output prints brackets verbatim (log lines,
vllmops command, hints likelitellm[proxy]), keeps copyable commands on one line, and sends diagnostics to stderr sovllmops proxy config > file.yamlis valid YAML - the TUI counts a model that belongs to several profiles once
vllmops doctorchecks the LiteLLM executable and flags models declared on the gateway port
Under the hood
- type checking with mypy strict and pyright on sources and tests
- CI on Python 3.10 through 3.14
Upgrade
uv tool upgrade vllmops # or: pipx upgrade vllmops
Existing .vllmops/config.yaml files keep working: the proxy section is optional and has defaults.