Skip to content

v1.4.0

Choose a tag to compare

@eladser eladser released this 29 Sep 22:00
· 1 commit to main since this release

Catches up with what the runners changed over the summer.

Fixed

  • vLLM 0.12 renamed its kv-cache metric, so the pane sat at 0%. Both names are read now.
  • Anything answering on a probed port (a dev server on :8000, say) no longer shows up as a phantom vllm or llama.cpp model.
  • Streamed OpenAI-style requests without a usage block now get recorded. llama.cpp's own timings are used when the stream carries them, otherwise tok/s is estimated from the chunk count and marked as such. The bytes your client gets are still untouched.
  • nvidia-smi [N/A] fields show as n/a instead of a fake 0, and stay out of the energy total.
  • A rocm-smi response mtop can't read surfaces as an error instead of a blank GPU pane.

New

  • The proxy records /v1/responses traffic, streamed or not.
  • llama.cpp router mode (llama-server started without -m) lists every loaded model.
  • LM Studio 0.4's v1 REST API, with a fallback for older installs.
  • A CHANGELOG, backfilled to 1.0.

brew and scoop pick this up as usual; winget follows once Microsoft merges the bump.