Skip to content

v0.5.2 — Docker image vLLM/CUDA fix + L40S VRAM/tok-s docs

Choose a tag to compare

View all tags
@cnygaard cnygaard released this 05 Jun 19:13
v0.5.2
7120cd3

v0.5.2

Patch release. The glq library is unchanged from v0.5.1 — this bump fixes the Docker image and adds measured perf docs.

Fixed

  • Docker image (ghcr.io/cnygaard/glq-env): install vLLM first so it resolves its own matching-CUDA torch (no torch pin / TORCH_INDEX_URL); base stays CUDA 12.8 to mirror the verified L40S venv. This fixes the broken :0.5.0/:0.5.1 images that failed at runtime with ImportError: libcudart.so.13 under --gpus all. The pip package was unaffected on all versions.

Docs

  • Checkpoints table now lists, per model: VRAM (resident weight footprint) and Tok/s (b1 / b32) — single-stream and 32-way batched decode — measured on a g6e.xlarge (NVIDIA L40S) via vLLM 0.20.2. E.g. SmolLM3-3B: 2.4 GiB, 35 / 654 tok/s.
  • New benchmarks/bench_tps_vllm.py two-point decode harness backing those numbers.
  • README notes the image-vLLM fix and points vllm serve at :latest.
pip install glq==0.5.2
Assets 2
Loading