dev (main)

Built from main, unreleased. Install it with:

vllm-helper upgrade --channel dev

commit: f991dd5

Merge pull request #51 from coder/devcats/hardware-detection

v0.5.0

Added

• Built-in docs. vllm-helper serve now has a Docs tab with a quickstart, the profile and memory knobs, the Coder integration, and a CLI reference generated from the command tree. The pages come from docs/, which stays readable on GitHub.
• Public docs site. cmd/vllm-helper-docs serves the same pages without the panel. It also serves each page's Markdown source, /llms.txt, /llms-full.txt and a sitemap. Dockerfile.docs deploys it on Railway. It is a separate binary, so vllm-helper itself gains no unauthenticated routes.
• Recipes: launch configurations that were run and measured, not inferred. Each one pins the model, the image (by digest), the exact argv, and the numbers it produced on named hardware.
  • recipes list and recipes show say where every setting came from.
  • recipes launch starts containers over SSH on the nodes you name; a recipe can span machines. It writes a profile of the same name and points Coder at what is serving, so the picker and the server always agree. Then it watches the load, like switch does.
  • recipes stop removes the containers.
  • recipes install-service installs a unit that keeps the active recipe serving, plus a timer that relaunches it when the endpoint stops answering.
    • It stands down the native unit, which can't serve a container.
    • It waits while a launch is in flight, so a loading model isn't mistaken for an outage and replaced with the previous profile.
    • A deliberate stop or systemctl stop stays stopped.
  • The web panel shows a recipe's health and log, but its buttons don't drive recipes yet; use the CLI.
• Runner-aware commands. status, logs, restart and stop act on whatever is actually serving. They used to assume the init-system service. That made them report systemd as healthy while a container served, or print "Stopped vllm.service" for a two-node cluster.
• Clean switches. Switching between a recipe and a native profile stops the one being left, so it frees the GPUs and port.
• Declared deviations. A recipe records each difference from its upstream recipe as data: the setting, upstream's value, ours, and why. tools/recipediff reports any undeclared difference. It compares content, not commit ids, so a pin that only trails upstream isn't reported as drift.
• More than one machine.
  • node add registers a machine once, by the URL it serves on and its SSH target. pool, shard and recipes launch then refer to it by name.
  • pool serve puts a router in front of replicas of one model and keeps each conversation on the node that already has its prefix cached. pool install-service runs the router as a unit.
  • shard start splits one model across machines with Ray or vLLM's multiprocessing executor, for trying a split before a recipe exists. shard install-ray prepares the nodes.
  • spark doctor and spark install-prereqs handle the build packages a DGX Spark needs.

Changed

• Coder entries are matched by model, not profile name. A renamed profile, or a second machine serving the same model, now adopts the existing entry instead of adding a duplicate row to the picker.
• doctor warns when the Coder token is within two days of expiring. An expired token silently stops picker updates.
• New setting CODER_MAX_OUTPUT_TOKENS. It raises Coder's server-side reply cap for this machine's models. If it's unset, nothing is sent and Coder's own setting stays.

v0.4.0

Added

• A web panel, vllm-helper serve: what's loaded and whether it's serving,
  memory in use against what this machine can give a model, the saved profiles
  with a switch button, and the server log. It can also search HuggingFace and
  add or remove models, measuring a result against this machine before saving.
  Control plane only — inference goes straight to vLLM. Localhost and a token
  unless --listen says otherwise.
• Choosing which quantization to serve, for GGUF repos that ship several:
  new --quant Q8_0, a prompt in search, a dropdown in the panel. Saved as
  MODEL=repo:QUANT with the --tokenizer such a model needs.
• install-gguf installs the plugin vLLM now needs to read GGUF at all, into
  the interpreter that owns the configured vLLM. Applies from the next switch.
• upgrade replaces the binary from a published release, checked against the
  release's checksum. --check reports without installing.
• A dev channel: upgrade --channel dev follows a build of main from every
  commit that passes CI; --channel stable returns to tagged releases.
• macOS (Apple Silicon): launchd, memory from sysctl, doctor checks the
  cmake/ninja/clang toolchain with brew hints.
• stop, restart, logs, inspect, edit, and set KEY=VALUE.
• switch, new, and restart watch the boot, report when the model is
  actually serving, and fail fast on a crash loop. --wait, --no-follow,
  --timeout.
• --json for status, list, search, and doctor.
• coder-sync prune clears Coder entries for models this machine can't serve.
  Reports by default, deletes with --yes.
• CODER_ORGANIZATION, for Coder deployments with more than one.
• version prints the commit, which is what tells two dev builds apart.
• doctor verifies Coder and HuggingFace credentials against their APIs.

Changed

• Long flags need two dashes. The old parser also accepted -json; the new
  one reads that as a bundle of short flags. logs -n still works.
• An argument too many is an error instead of being dropped, so
  vllm-helper status runing fails rather than reporting on something else.
• The KV cache is sized for the model being served instead of taking whatever
  vLLM finds left over. Every profile carried GPU_MEMORY_UTILIZATION=0.85, and
  that fraction is what decided the cache: a 1.1 GB model took 101.77 GiB of it.
  New profiles get KV_CACHE_MEMORY_BYTES, computed from the model's attention
  shape and KV_CONTEXT_TARGET full contexts (four by default), and
  --gpu-memory-utilization becomes a ceiling rather than the cache size.
  Existing profiles are untouched. A profile written here names a flag older
  vLLM builds don't have.
• The model budget is measured from the machine rather than fixed at 70% of
  unified memory or 80% of VRAM, both of which were invented — on a 128 GB
  machine the first held back 37 GiB for an OS that measures under 6 GiB. It is
  re-measured every run, so MODEL_BUDGET_GB is now an override and setup
  writes it commented out. If you ran setup on an earlier version, that
  number is still in your config.env and now pins the budget; comment it out.
  doctor says when an override is in force.
• inspect, search and the panel list every quant in a repo and lead with the
  biggest that fits, rather than reporting only the smallest — a 32B repo was
  being shown as 9.0 GB, which was its 2-bit quant. They also report how much
  context the model can hold on this machine.
• Quants vLLM can't load are marked, not offered: it addresses GGUF by name and
  only for a single file at the repo root, so split-file and subfolder copies
  are listed as unselectable, and a machine without the plugin is told before
  the pull rather than after the restart.
• GGUF builds of MoE models are marked unsupported. The plugin maps fused expert
  weights for OLMoE and MiniMax-M2 only; every other MoE fails at load, after
  the restart has already taken down what was serving.
• Coder sync is now a plugin behind a small interface. Nothing changes for
  users — same verbs, same config, same state file; it keeps the churn of an
  undocumented experimental API in one package.
• Commands are built on cobra: grouped
  --help, per-command flags, cobra's wording for unknown ones. Exit codes
  unchanged. --help now works on set, edit, remove, and coder-sync.
• The boot log banner says that Ctrl+C stops watching and leaves the server
  loading. People were interrupting a first-run download believing they had
  killed it.
• Non-interactive sessions never prompt, tail, or block on sudo, which makes
  the CLI usable from scripts and a UI backend.
• EXTRA_ARGS is split shell-style, so a quoted value with spaces survives.
• coder-sync register-all only backfills what's missing.
• start exits cleanly when no model is active, instead of sending
  Restart=on-failure into a crash loop with nothing to serve.

Fixed

• gpt-oss models work. Their cards carry no vllm serve snippet, so profiles
  were written with no flags, and served that way the model answers chat but
  returns an empty message for anything involving tools. new now adds the
  parsers and fetches the vocab files, where the installed vLLM confirms them.
• Model-card flags are checked against the vLLM installed here before reaching a
  profile. Anything this build won't accept used to be copied verbatim and
  surface as a crash loop on the next switch. Nothing is substituted
  automatically: parsers one character apart are different parsers.
• More of a card's flags are adopted — only five were recognised, so settings
  like --max-num-seqs were dropped without a word.
• --gpu-memory-utilization is capped where the GPU shares memory with the host
  (GB10, Jetson, Thor). The old guard asked whether an NVIDIA GPU was present,
  which is right for discrete VRAM and wrong for unified, where overshooting can
  wedge the host below the CUDA runtime rather than OOM cleanly. Linux hosts
  with no GPU are clamped to what's free, and no longer charged for a CUDA
  context they never create — a 7 GB CPU-only box was told nothing would run.
• Fit checks were 7.4% pessimistic: HuggingFace reports decimal GB and memory
  was read as GiB, so models that fit were called too big.
• Weights are measured wherever they live. The listing only looked at the repo
  root, so a model keeping weights one level down measured as 0 and read as
  "unknown", and a sharded repo past one page was never counted in full.
• Coder sync follows the model-config collection to
  /api/v2/organizations/{org}/chats/models. Against Coder 2.37 every sync
  failed with a 404 that read like a bad URL or token — which is what doctor
  said. Both generations stay supported.
• status tells a service that is failing to start from one still loading.
  Reporting a crash-looping unit as "still loading" sent people off to wait for
  a server that was never coming up; it now shows the last log line.
• The capability probe had stopped working wherever a vLLM plugin was installed:
  plugin registration on stdout meant the probe's JSON no longer parsed, so flag
  validation silently stopped and an installed GGUF plugin read as missing.
• inspect and the panel's model detail ran vllm --version before consulting
  the cache, which imports vLLM: 10.1s per lookup on a DGX Spark, now 0.25s.
• Installing the GGUF plugin from the panel no longer holds a request open for
  the whole source build. It runs on the server and the page polls it, so a
  reload rejoins an install in progress.
• new clamps MAX_MODEL_LEN to the model's max_position_embeddings even
  when the card has no snippet — the clamp sat behind an early return taken on
  exactly those cards, so facebook/opt-125m got 131072 and vLLM refused.
• set and edit refuse a MAX_MODEL_LEN that isn't a positive number, and
  set validates the whole batch before writing anything.
• A bad MAX_MODEL_LEN is reported before it reaches Coder, instead of becoming
  a context limit of 0 and an HTTP 400 that names nothing.
• Bare JSON in EXTRA_ARGS was mangled by shell-style splitting, turning
  {"thinking":true} into {thinking:true}, which vLLM rejects.
• Profile values round-trip: unusual leading or trailing whitespace was written
  unquoted and read back empty, and a NUL byte came back as a backslash.
• adopt saved a lossy copy of the server it took over — seven flags were
  recognised and the rest dropped, and --gpu-memory-utilization was hardcoded
  to 0.85, so a server tuned to 0.92 came back with less KV cache. It could
  also carry credential-bearing flags into a profile and into doctor --json.
• upgrade --check no longer claims an update when you're already on that
  channel, and no longer writes UPDATE_CHANNEL while reporting.
• --version prints the commit, matching version.
• Redirecting to /dev/null no longer looks like a terminal, so
  vllm-helper logs > /dev/null prints and exits instead of following forever.
• switch could wait forever: only a failed unit ended the wait, so a service
  that went inactive left it polling a health endpoint that never answered.
• doctor --yes hung on a sudo prompt with no terminal to answer it.
• status reports the host's service manager rather than always saying systemd.
• -- is honored as an end-of-flags terminator.

v0.3.1

Changelog

• b34738a Merge pull request #4 from coder/fix/chat-template-system-message-position-bpmct
• e4891dc fix: detect and patch chat templates that crash on misplaced system messages

v0.3.0

Changelog

• 5c66bd7 feat: thinking-mode flag detection and toolchain doctor (bash)
• d47be5c refactor!: Go binary is the sole implementation

v0.2.0

Changelog

• 801843c feat: Go CLI with detection/adoption engine and binary release pipeline