candle-vllm v0.9.1

What's Changed

• Fix nvfp4 scaling issue by @guoqingbao in #456
• Support Qwen3.8 models & quantized logits projection by @guoqingbao in #457
• Support more GGUF IQ format models by @guoqingbao in #458
• Reasoning effort support by @guoqingbao in #459
• Fix Qwen3.8 precision issue by @guoqingbao in #460

Full Changelog: v0.9.0...v0.9.1

candle-vllm v0.9.0

What's Changed

• Support new format AWQ/GPTQ MoE models by @guoqingbao in #452
• Align runtime and FP8 kernel paths by @guoqingbao in #453
• Support Multi-token Prediction (MTP) by @guoqingbao in #454

Full Changelog: v0.8.9...v0.9.0

candle-vllm v0.8.9

What's Changed

• Sharding improvement: KV-head replication by @guoqingbao in #449
• Improve batch capacity and graph capture by @guoqingbao in #450
• Rewrite tool calling logics & fix Gemma4 model precision issue by @guoqingbao in #451

Full Changelog: v0.8.8...v0.8.9

candle-vllm v0.8.8

What's Changed

• Fix Qwen3.5 models on Metal by @guoqingbao in #443
• Update Metal build OS target by @guoqingbao in #444
• Implement CPU offloading for kvcache and gdn states by @guoqingbao in #445
• Support CUDA 13.1, 13.2 and 13.3 by @guoqingbao in #446
• Optimize metal kernel path by @guoqingbao in #447
• Support MLX models & fix GLM-4.7 Flash model precision by @guoqingbao in #448

Full Changelog: v0.8.7...v0.8.8

candle-vllm v0.8.7

What's Changed

• Improve prebuild package compatibilities & add webdoc by @guoqingbao in #438
• Improve GGUF multirank precision and GDN prefix cache by @guoqingbao in #440
• Fix mamba slots release issue under aborted requests by @guoqingbao in #441
• Support DeepSeekV3.2/GLM5.2 models by @guoqingbao in #439
• Use Ubuntu 20.04 for SM70-SM89 build & fix benchmark bug by @guoqingbao in #442

Full Changelog: v0.8.6...v0.8.7

candle-vllm v0.8.6

Full Changelog: v0.8.5...v0.8.6

candle-vllm v0.8.5

What's Changed

• Revise server binding to support Unix sockets by @guoqingbao in #432
• Replace MPI with TCP coordinator for multinode inference by @guoqingbao in #434
• Update ReadMe by @guoqingbao in #435
• Improve GGUF model sharding by @guoqingbao in #436
• Improve streaming channels, tokenizer parsing, and GGUF MoE routing by @guoqingbao in #437

Full Changelog: v0.8.4...v0.8.5

candle-vllm v0.8.4

What's Changed

• Fix gguf exccessive memory usage & nvfp4 model regression by @guoqingbao in #431

Full Changelog: v0.8.3...v0.8.4

candle-vllm v0.8.3

What's Changed

• added flan-t5 example into test by @bm777 in #5
• Add PagedAttention by @EricLBuehler in #13
• Add working scheduler by @EricLBuehler in #15
• Add devcontainer by @sigma-andex in #17
• Fix pipeline generation (kernel launch, kernel compilation, rwlock, paged attention, etc.) by @guoqingbao in #37
• Correct generation with paged attention (fix kernel launch, kvcache, llama pipeline, etc.) by @guoqingbao in #38
• Dusting the project off by @EricLBuehler in #39
• Optional logprobs & fix llama eos/stop token by @guoqingbao in #40
• Configurable kvcache & fix repeat chat history by @guoqingbao in #41
• Support stream chat completion & optimization for decoding stage by @guoqingbao in #42
• Unified pipeline for models & support phi3 model by @guoqingbao in #45
• Support qwen2 model, optimize phi3 model, revise model loading strategy by @guoqingbao in #46
• Error prompt for requested message exceeds model capacity by @guoqingbao in #48
• Support Gemma model & remove repeat_kv (replaced with broadcast matmu… by @guoqingbao in #49
• Higher precision for rope in Gemma model. by @guoqingbao in #50
• Fix bug for previous removal of repeat_kv (when key_value_heads > 1 and < attention_heads) by @guoqingbao in #52
• Support Phi2 and Mistral models, fix generation remainder, more sampling parameters, etc. by @guoqingbao in #53
• Fix mistral model & more optional model-specific parameters. by @guoqingbao in #54
• Fix mistral output repetition with F32 rope and penalty & temperature parameters by @guoqingbao in #55
• Fix corner case when block table too small by @EricLBuehler in #56
• Support Yi & StableLM models, change default maximum length of generated tokens for smooth chat. by @guoqingbao in #57
• Fix build by @guoqingbao in #58
• Trim HF token by @EricLBuehler in #60
• Switch streaming service to axum & standalone generation thread by @guoqingbao in #63
• Fix typo & update ReadMe by @guoqingbao in #64
• Fix bug for non-streaming generation. by @guoqingbao in #65
• More elegant way for handing non-streaming finish signal. by @guoqingbao in #66
• LLaMa3.1 chat completion by @guoqingbao in #67
• Update demo video by @guoqingbao in #68
• Support streaming batched chat completion requests by @guoqingbao in #69
• Fix bug for token decoding & remove token padding by @guoqingbao in #71
• Fix bug for space token decoding & remove redundant code by @guoqingbao in #72
• Tweak sampling parameters & update batched generation results by @guoqingbao in #73
• Parallel token sampling process & reset decoder after each generation by @guoqingbao in #74
• Support in-situ quantization by @guoqingbao in #77
• Optimize quantized matmul in batch processing & update Q4K results by @guoqingbao in #78
• Fix bug for non-stream response by @guoqingbao in #80
• Ask users to provide huggingface token if no token cached and passed to the program. by @guoqingbao in #81
• No crash when both hidden_act and hidden_activation are set for gemma models by @guoqingbao in #82
• Apply clippy by @EricLBuehler in #83
• Restore previous bug fix by @guoqingbao in #85
• Support softcapping (Gemma-2 models) by @guoqingbao in #86
• Fix Gemma-2 multiple eos/bos ids by @guoqingbao in #87
• Custom benchmark with parameters by @guoqingbao in #88
• Support GPTQ/Marlin format quantization (4bit weight, f16 input) by @guoqingbao in #89
• Support bf16 inputs for GPTQ/Marlin format quantization by @guoqingbao in #90
• Add an example for marlin format conversion & update results by @guoqingbao in #91
• Support in-situ conversion of GPTQ model to marlin format (naive GPTQ kernel also supported). by @guoqingbao in #92
• Bump to candle v1.8.1 (cargo/rustc v1.83 required) by @guoqingbao in #95
• Making CUDA backend as a feature (prepared for Metal devices) by @guoqingbao in #96
• Support Mac/Metal devices by @guoqingbao in #97
• Display all simple chat output instantly & update GPU demo video by @guoqingbao in #98
• Fix kernel rebuild bugs by @guoqingbao in #100
• Support Multi-GPU inference on CUDA devices by @guoqingbao in #101
• Fix bug for simple chat demo & remove limits for max_gen_tokens by @guoqingbao in #103
• Guideline for running DeepSeek-R1 distilled models by @guoqingbao in #104
• Use chat template from config file(s) by @guoqingbao in #105
• Support DeepSeek models by @guoqingbao in #107
• Optimize deepseek model by @guoqingbao in #108
• Faster token sampling by @guoqingbao in #109
• Support distributed inference of deepseek models by @guoqingbao in #110
• Fix chat template problem & prepare all models for multi-gpu inference by @guoqingbao in #111
• Support distributed inference of quantized models by @guoqingbao in #114
• Fix StopTokens deserialization by @leejuyuu in #113
• Enable custom stop token obtained from request parameters by @guoqingbao in #115
• Support Qwen2 (QwQ-32B) GGUF model by @guoqingbao in #116
• Fix bug for apply_chat_template by @guoqingbao in #117
• Improve performance for parallel token sampling by @guoqingbao in #118
• Configurable holding time for parallel process & Fix bug for non-sharded qlinear by @guoqingbao in #119
• Improve parallel inference with thread sync by @guoqingbao in #120
• Support multiprocess multi-gpu inference by @guoqingbao in #121
• Enable heartbeat detection for multiprocess inference by @guoqingbao in #122
• Tweak command channel connection by @guoqingbao in #123
• Improve performance of deepseek models (Rust compiler 1.83.0+ required) by @guoqingbao in #124
• Fix bug for group topk in deepseek v2 by @guoqingbao in #125
• Fix process sync issue when chat streaming aborted by @guoqingbao in #126
• Model loading progressbar & configurable log by @guoqingbao in #127
• Print maximum model length for different batch selection by @guoqingbao in #128
• Fix bug for progressbar & ensure all ranks finish loading before starting service by @guoqingbao in #129
• Support AWQ quant & DeepSeek-R1 685B on less GPU memory (CPU offloading) by @guoqingbao in #130
• Fix bug for sharded gptq model by @guoqingbao in #131
• Support multi-node inference with mpi runner by @guoqingbao in https://github.com/E...