v0.6.7

What's Changed

• Core: add output streaming support to multi-step + async by @AlpinDale in #1112
• tests: update scheduler tests by @AlpinDale in #1113
• (1/N) XQA: integrate the XQA CUDA kernels within Aphrodite by @AlpinDale in #1115
• chore: support loading weights by ID within models by @AlpinDale in #1116
• chore: expose phi3_v num_crops as an mm_processor_kwargs by @AlpinDale in #1117
• fix: unsafe all-reduce sync by @AlpinDale in #1118
• kernels: split marlin kernels for faster compile, fix MoE, temporarily remove HQQ by @AlpinDale in #1119
• LLM: enable batched inference for llm.chat() API by @AlpinDale in #1120
• Quantization: re-enable Marlin serialization for AWQ quants by @AlpinDale in #1121
• fix: torch.compile dynamo fix by @AlpinDale in #1122
• chore: bump bitsandbytes version to latest; enable cuda graphs for 4bit bnb by @AlpinDale in #1123
• (1/N) Triton Backend: integrate Triton layernorm kernels by @AlpinDale in #1125
• (2/N) Triton Backend: integrate Triton activation kernels by @AlpinDale in #1126
• chore: remove trailing whitespaces by @AlpinDale in #1128
• chore: support prompt_logprobs with speculative decoding by @AlpinDale in #1129
• feat: add Priority-based Scheduling by @AlpinDale in #1130
• API: use heartbeats instead of health checks by @AlpinDale in #1131
• kernel: fix custom all-reduce kernel compilation on Pascal GPUs by @AlpinDale in #1132
• fix: propagate trust_remote_code in InternVL and MiniCPM-V by @AlpinDale in #1133
• fix: load fully-connected layer bias for EAGLE models by @AlpinDale in #1134
• API: propagate usage accounting to FastAPI middleware layer by @AlpinDale in #1135
• fix: ray 2.9.x does not expose available_resources_per_node by @AlpinDale in #1136
• fix: multi-step scheduling with InternVL by @AlpinDale in #1137
• chore: support FP8 MoE for compressed-tensors by @AlpinDale in #1138
• model: add support for Mllama (Llama 3.2) models by @AlpinDale in #1139
• fix: quantization for Mllama models by @AlpinDale in #1140
• fix: include encoder prompt len to non-stream api usage response by @AlpinDale in #1141
• fix: downgrade logger.warning for BOS fallback to print_warning_once by @AlpinDale in #1142
• fix: only set tool_choice to auto if at least one tool is provided by @AlpinDale in #1143
• API: add tool calling support for Llama 3.1 and 3.2 by @AlpinDale in #1144
• fix: batched inference with fuyu by @AlpinDale in #1145
• TPU: support Trillium by @AlpinDale in #1146
• torch.compile: use empty tensor instead of None for profiling by @AlpinDale in #1147
• kernel: Integrate asymmetric quantization for INT8 activations by @AlpinDale in #1148
• core: add support for chunked prefill + multi-step scheduling by @AlpinDale in #1149
• distributed: add env var to force custom all-reduce by skipping p2p check by @AlpinDale in #1150
• chore: add priority scheduling to async engine by @AlpinDale in #1151
• fix: XPU docker build by @AlpinDale in #1153
• distributed: force full nvlink when APHRODITE_FORCE_P2P env var by @AlpinDale in #1154
• fix: multi-step scheduling with Pipeline Parallel by @AlpinDale in #1155
• chore: improve implicit choice of spawn/fork for multiprocessing method by @AlpinDale in #1156
• fix: block manager v2 with preemption and lookahead slots by @AlpinDale in #1157
• fix: marlin MoE act order when is_k_full==False by @AlpinDale in #1158
• build: set FETCHCONTENT_BASE_DIR to one location for better caching by @AlpinDale in #1159
• model: add support for Qwen2.5-Math-RM-72B reward model by @AlpinDale in #1160
• lora: add LoRA support for MiniCPMV-2.5 by @AlpinDale in #1161
• fix: seeded gens with encoder-decoder models by @AlpinDale in #1162
• api: add support for prefill to chat completions endpoint by @AlpinDale in #1163
• kernel: varlen prefill + prefill chunking support for mamba kernels by @AlpinDale in #1164
• model: support input embeddings for qwen2-vl by @AlpinDale in #1165
• models: add LoRA support for MiniCPM-V 2.6 by @AlpinDale in #1166
• vlm: expose internvl2 max_dynamic_patch as a mm_processor_kwarg by @AlpinDale in #1167
• api: expose priority scheduling in the API server by @AlpinDale in #1168
• feat: add request-level logging by @AlpinDale in #1169
• fix: adjust max_position_embeddings for LoRA by @AlpinDale in #1170
• core: move guided decoding params into sampling params by @AlpinDale in #1171
• build: fix machete generation file ordering by @AlpinDale in #1172
• fix: torch.compile tensor alias by @AlpinDale in #1173
• chore: add process_weights_after_loading for DummyLoader by @AlpinDale in #1174
• fix: tensor-parallel inference with fuyu by @AlpinDale in #1175
• fix: token IDs reference fro MiniCPM-V when images are provided with no placeholders by @AlpinDale in #1176
• (1/N) MQA Scorer: add MQA scorer by @AlpinDale in #1177
• feat: support multi-step, chunked prefill, prefix cache, cuda graph combo by @AlpinDale in #1178
• fix: guided decoding default values breaking text completions API by @AlpinDale in #1179
• OpenVINO: add support for GPU, fix Docker build by @AlpinDale in #1181
• models: add support for Granite MoE model (PowerMoE) by @AlpinDale in #1182
• fix: mistral parallel tool call template fix by @AlpinDale in #1183
• fix: enforce mistral tool call ID constraint by @AlpinDale in #1184
• core: make v2 block manager the default by @AlpinDale in #1185
• logging: only log the non-default parameters in engine by @AlpinDale in #1180
• chore: parse literals out of --override-neuron-config by @AlpinDale in #1186
• [torch.compile]: add forward context for attention by @AlpinDale in #1187
• [torch.compile]: add forward context for flashinfer by @AlpinDale in #1188
• fix: OPT model loading for checkpoints with no tied embeds by @AlpinDale in #1189
• api: add tool parser plugin + InternLM2 tool parser by @AlpinDale in #1190
• build: only build kernels supported by current GPU by @AlpinDale in #1191
• chore: enable multi_step_stream_outputs by default by @AlpinDale in #1192
• feat: add support for xgrammar + GBNF grammar by @AlpinDale in #1193
• Revert "feat: add support for xgrammar + GBNF grammar (#1193)" by @AlpinDale in #1194
• feat: add support for xgrammar + GBNF grammars by @AlpinDale in #1195
• sampler: add option to ban tokens within a specific range of output tokens by @AlpinDale in #1196
• models: add pipeline parallel support for all models by @AlpinDale in #1197
• fix: reshape the dimensions of the input image embeds in Qwen2VL by @AlpinDale in #1198
• quantization: add VPTQ support by @AlpinDale in #1199
• api: enable prompt truncation for embeddings API by @AlpinDale in #1200
• MoE: add FusedMoE support for gptq_marlin and awq_marlin by @AlpinDale in #1201
• neuron: add on-device sampling support by @AlpinDale in #1202
• torch.compile: improve all-reduce registration by @AlpinDale in #1203
• fix: use block manager v1 for encoder-decoder by @AlpinDale in #1204
• fix: phi3v and ultravox multimodal EmbeddingInputs support by @AlpinDale in #1205
• models: add support for Gemma2 Embedding model by @AlpinDale in #1206
• fix: deprecate registration of custom configs to HF by @AlpinDale in #1207
• distributed: add pipeline parallel support for embedding models by @AlpinDale in #1208
• fix: incorrect updates to num_computed_tokens in multistep scheduling by @AlpinDale in #1209
• cpu: fix model input for decode by @AlpinDale in #1210
• fix: block manager v2 with encoder-decoder models by @AlpinDale in #1212
• models: explicit interface for model registry by @AlpinDale in #1213
• CPU: cross-attention and encoder-decoder model support by @AlpinDale in #1214
• quantization: optimize GGUF config loading and refactor parameter packing and forwarding by @AlpinDale in #1215
• gguf: fix qwen config loading by @AlpinDale in #1216
• VLM: add support for NVLM-D model by @AlpinDale in #1217
• XPU: fix XPU decode input by @AlpinDale in #1218
• VLM: add support for runtime mm_processor_kwargs by @AlpinDale in #1219
• fix: handle older torch versions in fake ops registration by @AlpinDale in #1220
• quantization: add bitsandbytes support to more models by @AlpinDale in #1221
• fix: remap FP8 kv_scale for command-r and DBRX by @AlpinDale in #1222
• fix: optimize composite weight loading & fix EAGLE weight loading by @AlpinDale in #1223
• fix: LoRA loading for compressed tensors by @AlpinDale in #1224
• fix: support loading LoRA weights for target_modules in pytorch bin by @AlpinDale in #1225
• fix: MiniCPM LoRA loading by @AlpinDale in #1226
• LoRA: add support for Qwen models by @AlpinDale in #1227
• lora: replace hardcoded cuda device with configurable arg by @AlpinDale in #1228
• lora: add support for bias tensors in LoRAs by @AlpinDale in #1229
• fix: fully sharded lora bug by @AlpinDale in #1230
• triton: explicitly specify other in tl.load by @AlpinDale in #1231
• lora: register punica ops directly for better eager mode performance by @AlpinDale in #1232
• fix: lora weight sharding in ColumnParallelLinearWithLoRA by @AlpinDale in #1233
• lora: move the bias implementation to punica.py by @AlpinDale in #1234
• lora: clean up the punica function interface by @AlpinDale in #1235
• lora: abstract away the punica wrapper by @AlpinDale in #1236
• fix: access get_vocab instead of vocab in tool parsers by @AlpinDale in #1237
• CPU: add support for AWQ quants (slow) by @AlpinDale in #1238
• fix: invalid args passed to _process_request in encode() by @AlpinDale in #1239
• chore: improve model support check in registry by @AlpinDale in #1241
• [BUG FIX] fix bugs to load llama 405B by vptq quant method by @wejoncy in #1240
• fix: machete kernels outputting garbage results for large K dim models by @AlpinDale in #1242
• fix: llama with tied weights + lora by @AlpinDale in #1243
• VLM: add image embedding input support for MiniCPM-V by @AlpinDale in #1244
• torch.compile: integrate aphrodite with torch.compile by @AlpinDale in #1246
• torch.compile: generic decorators by @AlpinDale in #1247
• chore: hide best_of parameter from the engine by @AlpinDale in #1248
• chore: refactor model registry and collect info in a single process by @AlpinDale in #1249
• fix: propagate priority through multiprocessing engine by @AlpinDale in #1250
• models: refactor mamba/attention-free models support by @AlpinDale in #1251
• VLM: add support for GLM-4V model by @AlpinDale in #1252

New Contributors

• @wejoncy made their first contribution in #1240

Full Changelog: v0.6.6...v0.6.7