One GPU, many model services — switch safely.

English | 简体中文

---

What it does

llmctl manages local model services on a GPU that can't hold them all at once. One YAML registry describes your services; llmctl handles the rest:

• Mutex switching — use stops conflicting services and starts the target. heavy services are globally exclusive; light ones may coexist.
• Safe teardown — stop script → port-release wait → SIGTERM → SIGKILL → VRAM-drain confirmation via nvidia-smi. Any failure aborts the switch instead of stacking one heavy service on top of another.
• Busy guard — an optional per-service probe (any shell command; exit 0 = working) prevents stopping a service mid-task, unless --force is given.
• Health probing — a service counts as started only when its health endpoint returns 200.
• Speed benchmark — bench measures decode and prefill tok/s.
• Power governance — optional per-service power limits applied automatically on switch (calibration SOP in docs/SOP_POWER.md).

No daemon, no state file: every run measures reality — ports, HTTP, nvidia-smi, pgrep.

Quick start

1. Install uv, clone this repo, then:

uv tool install .                                    # installs the llmctl command
mkdir -p ~/.config/llmctl
cp registry.example.yaml ~/.config/llmctl/registry.yaml   # edit for your services
llmctl status                                        # verify service detection

Tip

For development inside the repo, uv run llmctl ... works too. Config resolution: $LLMCTL_REGISTRY → ./registry.yaml → ~/.config/llmctl/registry.yaml.

2. (Optional) power features — passwordless sudo scoped to nvidia-smi -pl:

echo "$USER ALL=(ALL) NOPASSWD: /usr/bin/nvidia-smi -pl *" | sudo tee /etc/sudoers.d/nvidia-smi-power

3. (Optional) agent skill — the repo ships an Agent Skills-compatible skill at .agents/skills/llmctl (mirrored to .claude/skills for Claude Code). Codex, opencode, Claude Code, Gemini CLI and other compatible agents discover it automatically; AGENTS.md points the rest to it.

Commands

Command	Description
llmctl status	Show process/health per service and GPU VRAM
llmctl use <svc...>	Switch to services (stops mutex services, applies power limit); --force overrides the busy guard
llmctl stop <svc...>	Stop services; --force overrides the busy guard; stop all also restores the default power limit
llmctl bench <svc>	Measure decode/prefill tok/s (service must be online and exclusive)
llmctl power <watts>	Set GPU power limit

Registering a service

Each service is one block in registry.yaml; registry.example.yaml documents every field. Required: port, health. Strongly recommended: group, vram_mb, kill_pattern (missing kill_pattern disables the kill escalation path; missing vram_mb skips the VRAM-drain check). The health endpoint must return 200 only when the service is truly ready (model loaded), and kill_pattern must be verified with pgrep -af. See .agents/skills/llmctl/SKILL.md for the full registration checklist.

Power tuning

Optional. docs/SOP_POWER.md walks through the calibration: bench at each power rung, keep the lowest one with ≥ 90% speed, write it into power_limit_w.

Requirements

• Linux + NVIDIA GPU (nvidia-smi)
• Python ≥ 3.10, uv
• Optional: passwordless sudo for nvidia-smi -pl (power features)