karmic-kraken-20261020

Wheel-built runtime: karmic-kraken / karmic-kraken

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-20261020

Digest: ghcr.io/local-inference-lab/vllm@sha256:94887a58d4c003a06cb2d96ee5522eb7cea216838cfdad318076f9a08e2e762c

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-20261019.2

Correctness and reliability

• DeepSeek-V4.1-Flash serves top_p 1.0, the sampling the DeepSeek-V4.1-Flash-1017 model card recommends (docker #161; by @voipmonitor)
  • Models: DeepSeek-V4.1-Flash
  • Compatibility: The server default top_p of PROFILE=ds41-flash changes from 0.95 to 1.0; temperature stays 1.0. Requests that send their own top_p are unaffected. Overriding override-generation-config with {temperature 1.0, top_p 0.95}, as any profile value is overridden, restores the previous default.
  • The profile carried override-generation-config {temperature 1.0, top_p 0.95} from its source launcher. The first DeepSeek-V4.1-Flash model card recommended top_p 0.95 or 1.0; the DeepSeek-V4.1-Flash-1017 card, served since docker-160, recommends top_p 1.0 only.
  • The 1017 checkpoints ship no generation_config.json and the profile keeps generation-config vllm, so this override is the only source of the server's sampling defaults.
  • parameter-docs and the runtime README describe the DeepSeek-V4.1-Flash defaults; GLM, DeepSeek-V4 and MiMo keep temperature 1.0, top_p 0.95.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	6023d031be8e32b4b8a749dbb5d17d7d246b939b	flashinfer-cu134-sm120-beta-6023d031be8e32b4b8a749dbb5d17d7d246b939b
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	dde2c874b3aa3ecb0e606d9e41a089f983cf35d8	lmcache-cu134-sm120-beta-dde2c874b3aa3ecb0e606d9e41a089f983cf35d8
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	8244d1c44348ae8a5bbfc9f1b607021652283ed4	vllm-jovian-cu134-beta-8244d1c44348ae8a5bbfc9f1b607021652283ed4

karmic-kraken-beta-20261020

Wheel-built runtime: karmic-kraken / karmic-kraken-beta

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261020

Digest: ghcr.io/local-inference-lab/vllm@sha256:bc48d33ee167fb7675a59889fe0a7fea539bc6700c53cada89284d594a754b54

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261019.2

Correctness and reliability

• DeepSeek-V4.1-Flash serves top_p 1.0, the sampling the DeepSeek-V4.1-Flash-1017 model card recommends (docker #161; by @voipmonitor)
  • Models: DeepSeek-V4.1-Flash
  • Compatibility: The server default top_p of PROFILE=ds41-flash changes from 0.95 to 1.0; temperature stays 1.0. Requests that send their own top_p are unaffected. Overriding override-generation-config with {temperature 1.0, top_p 0.95}, as any profile value is overridden, restores the previous default.
  • The profile carried override-generation-config {temperature 1.0, top_p 0.95} from its source launcher. The first DeepSeek-V4.1-Flash model card recommended top_p 0.95 or 1.0; the DeepSeek-V4.1-Flash-1017 card, served since docker-160, recommends top_p 1.0 only.
  • The 1017 checkpoints ship no generation_config.json and the profile keeps generation-config vllm, so this override is the only source of the server's sampling defaults.
  • parameter-docs and the runtime README describe the DeepSeek-V4.1-Flash defaults; GLM, DeepSeek-V4 and MiMo keep temperature 1.0, top_p 0.95.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	79819b112f8bda7d09709c24bb31de16d7ce59c6	flashinfer-cu134-sm120-beta-79819b112f8bda7d09709c24bb31de16d7ce59c6
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	dde2c874b3aa3ecb0e606d9e41a089f983cf35d8	lmcache-cu134-sm120-beta-dde2c874b3aa3ecb0e606d9e41a089f983cf35d8
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db	vllm-jovian-cu134-beta-6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db

karmic-kraken-20261019.2

Wheel-built runtime: karmic-kraken / karmic-kraken

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-20261019.2

Digest: ghcr.io/local-inference-lab/vllm@sha256:21ed89d039083a2b9db582cf91dc9689e087c728210e19b52b05cb24a4c84954

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-20261019

Features

• DeepSeek-V4.1-Flash serves DeepSeek's 2026-10-17 release (DeepSeek-V4.1-Flash-1017) (docker #160; by @voipmonitor)
  • Models: DeepSeek-V4.1-Flash
  • Compatibility: The first start downloads the new checkpoint (about 495 GB for the FP4-CSF default); LMCache disk tiers written with the previous checkpoint are not reused.
  • PROFILE=ds41-flash serves local-inference-lab/DeepSeek-V4.1-Flash-1017-lossless-CSF (revision 90d665c1), the lossless FP4-CSF container of deepseek-ai/DeepSeek-V4.1-Flash-1017. CHECKPOINT=original serves deepseek-ai/DeepSeek-V4.1-Flash-1017 (revision 1bae0e10).
  • Loader, quantization, sampling defaults and all other profile values are unchanged.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	6023d031be8e32b4b8a749dbb5d17d7d246b939b	flashinfer-cu134-sm120-beta-6023d031be8e32b4b8a749dbb5d17d7d246b939b
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	dde2c874b3aa3ecb0e606d9e41a089f983cf35d8	lmcache-cu134-sm120-beta-dde2c874b3aa3ecb0e606d9e41a089f983cf35d8
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	8244d1c44348ae8a5bbfc9f1b607021652283ed4	vllm-jovian-cu134-beta-8244d1c44348ae8a5bbfc9f1b607021652283ed4

karmic-kraken-beta-20261019.2

Wheel-built runtime: karmic-kraken / karmic-kraken-beta

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261019.2

Digest: ghcr.io/local-inference-lab/vllm@sha256:b3dcc15430e002cb3294e8ab14553ca5f3922e6b8c89507514cd160b04a36c55

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261019

Features

• DeepSeek-V4.1-Flash serves DeepSeek's 2026-10-17 release (DeepSeek-V4.1-Flash-1017) (docker #160; by @voipmonitor)
  • Models: DeepSeek-V4.1-Flash
  • Compatibility: The first start downloads the new checkpoint (about 495 GB for the FP4-CSF default); LMCache disk tiers written with the previous checkpoint are not reused.
  • PROFILE=ds41-flash serves local-inference-lab/DeepSeek-V4.1-Flash-1017-lossless-CSF (revision 90d665c1), the lossless FP4-CSF container of deepseek-ai/DeepSeek-V4.1-Flash-1017. CHECKPOINT=original serves deepseek-ai/DeepSeek-V4.1-Flash-1017 (revision 1bae0e10).
  • Loader, quantization, sampling defaults and all other profile values are unchanged.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	79819b112f8bda7d09709c24bb31de16d7ce59c6	flashinfer-cu134-sm120-beta-79819b112f8bda7d09709c24bb31de16d7ce59c6
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	dde2c874b3aa3ecb0e606d9e41a089f983cf35d8	lmcache-cu134-sm120-beta-dde2c874b3aa3ecb0e606d9e41a089f983cf35d8
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db	vllm-jovian-cu134-beta-6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db

karmic-kraken-beta-20261019

Wheel-built runtime: karmic-kraken / karmic-kraken-beta

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261019

Digest: ghcr.io/local-inference-lab/vllm@sha256:376745af9ebffe0ecb3a9b7ee3d2040027834e58e3248898e52487255f33125a

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261016.2

Correctness and reliability

• Recurrent checkpoint restores no longer recompute a prompt because a rank's transfer worker rejected an admitted copy (lmcache #113; by @ktsaou)
  • Models: GLM-5.3-Flash, Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash, DeepSeek-V4-Flash
  • The recurrent checkpoint bridge admits up to 32 stores and restores at a time, but each TP rank's transfer worker was created with room for 16. Under copy pressure a healthy worker rejected a copy the scheduler had already admitted; the connector reported it as a miss, a restore spent another lookup attempt and could end up recomputing its prompt without reading the payload on that rank.
  • Each rank's worker now admits 34 copies (the 32 the scheduler can hand it plus one completed job per copy thread whose permit is released a moment after its result is visible), with two copy threads as before.
  • Six CPU regressions exercise admission of the full scheduler allowance on both ranks with delayed completion callbacks.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	79819b112f8bda7d09709c24bb31de16d7ce59c6	flashinfer-cu134-sm120-beta-79819b112f8bda7d09709c24bb31de16d7ce59c6
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	dde2c874b3aa3ecb0e606d9e41a089f983cf35d8	lmcache-cu134-sm120-beta-dde2c874b3aa3ecb0e606d9e41a089f983cf35d8
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db	vllm-jovian-cu134-beta-6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db

karmic-kraken-beta-20261016

Wheel-built runtime: karmic-kraken / karmic-kraken-beta

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261016

Digest: ghcr.io/local-inference-lab/vllm@sha256:350e6134541cf4cbf41f04ab4cf590fe0c74e0d865ff70a18eafd3deabbf74bc

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261015.4

Features

• --print-command prints the preset's vllm serve process as a docker run script (docker #158; by @voipmonitor)
  • Models: all
  • Compatibility: No user action required; nothing changes for a normal start.
  • docker run --rm -e PRESET= --print-command prints the vllm serve process behind the preset as a shell script: a docker run command with every environment variable the launcher sets and the vLLM options as the container command, which keeps the image's CUDA/NCCL bootstrap and skips the profile launcher. IMAGE= GPUS=0,1 PORT=8000 bash <script> starts it.
  • FP4-CSF checkpoints in the Hugging Face layout are named by repository and --revision with the execution-time options (--quantization modelopt_mixed --load-format instanttensor), so vLLM downloads the pinned revision itself. Presets with the external LMCache tier print their CACHE_MODE=vram form.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	79819b112f8bda7d09709c24bb31de16d7ce59c6	flashinfer-cu134-sm120-beta-79819b112f8bda7d09709c24bb31de16d7ce59c6
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	8ef537d6481b0d7c93277118ffcc6bf7e68d1b22	lmcache-cu134-sm120-beta-8ef537d6481b0d7c93277118ffcc6bf7e68d1b22
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db	vllm-jovian-cu134-beta-6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db

karmic-kraken-beta-20261016.2

Wheel-built runtime: karmic-kraken / karmic-kraken-beta

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261016.2

Digest: ghcr.io/local-inference-lab/vllm@sha256:f5f9efe1b10ec7b853a495e66b7dbf15ff54f62741e444dea0b44e16736a10a9

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-beta-20261016

Correctness and reliability

• --print-command also works for deployments with an explicit external cache (docker #159; by @voipmonitor)
  • Models: all
  • --print-command refused CACHE_MODE=lmcache / --cache-mode lmcache given explicitly, which the configurator's generated commands always carry. It now prints such a deployment in its CACHE_MODE=vram form (GPU prefix cache only, same KV size), as it already did for the presets' default external cache, and says so in the script header.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	79819b112f8bda7d09709c24bb31de16d7ce59c6	flashinfer-cu134-sm120-beta-79819b112f8bda7d09709c24bb31de16d7ce59c6
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	8ef537d6481b0d7c93277118ffcc6bf7e68d1b22	lmcache-cu134-sm120-beta-8ef537d6481b0d7c93277118ffcc6bf7e68d1b22
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db	vllm-jovian-cu134-beta-6d66eed35b0ab1cc0e05f00e83ef638ba10ae3db

karmic-kraken-20261016

Wheel-built runtime: karmic-kraken / karmic-kraken

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-20261016

Digest: ghcr.io/local-inference-lab/vllm@sha256:c14295c214f6126f0fc362f3e6073933a2125de9fc35b938e43842a5dcc40e38

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-20261015.8

Features

• --print-command prints the preset's vllm serve process as a docker run script (docker #158; by @voipmonitor)
  • Models: all
  • Compatibility: No user action required; nothing changes for a normal start.
  • docker run --rm -e PRESET= --print-command prints the vllm serve process behind the preset as a shell script: a docker run command with every environment variable the launcher sets and the vLLM options as the container command, which keeps the image's CUDA/NCCL bootstrap and skips the profile launcher. IMAGE= GPUS=0,1 PORT=8000 bash <script> starts it.
  • FP4-CSF checkpoints in the Hugging Face layout are named by repository and --revision with the execution-time options (--quantization modelopt_mixed --load-format instanttensor), so vLLM downloads the pinned revision itself. Presets with the external LMCache tier print their CACHE_MODE=vram form.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	6023d031be8e32b4b8a749dbb5d17d7d246b939b	flashinfer-cu134-sm120-beta-6023d031be8e32b4b8a749dbb5d17d7d246b939b
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	8ef537d6481b0d7c93277118ffcc6bf7e68d1b22	lmcache-cu134-sm120-beta-8ef537d6481b0d7c93277118ffcc6bf7e68d1b22
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	8244d1c44348ae8a5bbfc9f1b607021652283ed4	vllm-jovian-cu134-beta-8244d1c44348ae8a5bbfc9f1b607021652283ed4

karmic-kraken-20261019

Wheel-built runtime: karmic-kraken / karmic-kraken

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-20261019

Digest: ghcr.io/local-inference-lab/vllm@sha256:3d3591647173bf807307eb9580f51df9bbe304624785191cfe8303732edcd4e7

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-20261016.2

Correctness and reliability

• Recurrent checkpoint restores no longer recompute a prompt because a rank's transfer worker rejected an admitted copy (lmcache #113; by @ktsaou)
  • Models: GLM-5.3-Flash, Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash, DeepSeek-V4-Flash
  • The recurrent checkpoint bridge admits up to 32 stores and restores at a time, but each TP rank's transfer worker was created with room for 16. Under copy pressure a healthy worker rejected a copy the scheduler had already admitted; the connector reported it as a miss, a restore spent another lookup attempt and could end up recomputing its prompt without reading the payload on that rank.
  • Each rank's worker now admits 34 copies (the 32 the scheduler can hand it plus one completed job per copy thread whose permit is released a moment after its result is visible), with two copy threads as before.
  • Six CPU regressions exercise admission of the full scheduler allowance on both ranks with delayed completion callbacks.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	6023d031be8e32b4b8a749dbb5d17d7d246b939b	flashinfer-cu134-sm120-beta-6023d031be8e32b4b8a749dbb5d17d7d246b939b
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	dde2c874b3aa3ecb0e606d9e41a089f983cf35d8	lmcache-cu134-sm120-beta-dde2c874b3aa3ecb0e606d9e41a089f983cf35d8
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	8244d1c44348ae8a5bbfc9f1b607021652283ed4	vllm-jovian-cu134-beta-8244d1c44348ae8a5bbfc9f1b607021652283ed4

karmic-kraken-20261016.2

Wheel-built runtime: karmic-kraken / karmic-kraken

Image: ghcr.io/local-inference-lab/vllm:karmic-kraken-20261016.2

Digest: ghcr.io/local-inference-lab/vllm@sha256:faf0b1f08d688b1aaa7d5161dc3b724c701d17d68278e7d32ebdd3a6b5127ce2

Native GPU smoke and LMCache checkpoint/filesystem contract tests; model-serving performance and full GLM cache E2E remain unqualified.

Changes since ghcr.io/local-inference-lab/vllm:karmic-kraken-20261016

Correctness and reliability

• --print-command also works for deployments with an explicit external cache (docker #159; by @voipmonitor)
  • Models: all
  • --print-command refused CACHE_MODE=lmcache / --cache-mode lmcache given explicitly, which the configurator's generated commands always carry. It now prints such a deployment in its CACHE_MODE=vram form (GPU prefix cache only, same KV size), as it already did for the presets' default external cache, and says so in the script header.

Reproducible component inputs

Component	Source commit	Wheel release
flashinfer	6023d031be8e32b4b8a749dbb5d17d7d246b939b	flashinfer-cu134-sm120-beta-6023d031be8e32b4b8a749dbb5d17d7d246b939b
instanttensor	747390dc2bd1e0a18b0c7b13a09a38594edf342f	instanttensor-cu134-beta-747390dc2bd1e0a18b0c7b13a09a38594edf342f
lmcache	8ef537d6481b0d7c93277118ffcc6bf7e68d1b22	lmcache-cu134-sm120-beta-8ef537d6481b0d7c93277118ffcc6bf7e68d1b22
nccl	300f3a73caba7196e436571d9ac34069e81edea4	nccl-cu134-sm120-300f3a73caba7196e436571d9ac34069e81edea4
vllm	8244d1c44348ae8a5bbfc9f1b607021652283ed4	vllm-jovian-cu134-beta-8244d1c44348ae8a5bbfc9f1b607021652283ed4