v0.6.6

What's Changed

• distributed: support pipeline parallelism for internvl and internlm2 by @AlpinDale in #965
• tpu: add support for async postprocessing by @AlpinDale in #968
• fix: prometheus.yaml path in monitoring example by @AlpinDale in #969
• tpu: support single and multi-host TPUs on GKE and RayServe by @AlpinDale in #970
• vlm: add tensor parallel support for vision transformer models by @AlpinDale in #971
• tests: update internvl test for #971 by @AlpinDale in #972
• vlm: increase the default max_num_batched_tokens for multimodal models by @AlpinDale in #973
• core: fix chunked prefill not being enabled by default for long contexts by @AlpinDale in #974
• tpu: fix TPU type api by @AlpinDale in #975
• fix: modelscope for VLMs by @AlpinDale in #976
• fix: crash when cancelling a request with multi-step by @AlpinDale in #977
• models: add support for IBM Granite (PowerLM) models by @AlpinDale in #978
• tpu: align worker index with node boundary by @AlpinDale in #979
• fix: InternLM2 model with Tensor Parallel by @AlpinDale in #980
• core: slightly improve chunked prefill performance by @AlpinDale in #981
• vlm: fallback to SDPA for ViT models on CPU backend by @AlpinDale in #982
• core: improve async postproc + multi-step performance by @AlpinDale in #983
• fix: raise exception when accessing logger for disable_log_stats=True case by @AlpinDale in #984
• chore: rename task_handler to worker by @AlpinDale in #985
• tpu: fix outputs by correcting the next_token_ids shape by @AlpinDale in #986
• quants: add GPTQ and FBGEMM to AphroditeParameters by @AlpinDale in #987
• benchmarks: add --async-engine arg to throughput benchmark by @AlpinDale in #988
• tpu: use XLA rank for persistent cache path by @AlpinDale in #989
• vlm: support multiple audios per prompt for Ultravox by @AlpinDale in #990
• vlm: fix siglip layernorm and paligemma weight loading by @AlpinDale in #991
• vlm: enable multimodal inputs for the LLM class by @AlpinDale in #992
• api: implement OpenAI-compatible tools API for Hermes/Mistral models by @AlpinDale in #993
• neuron: add 8bit quantization for Neuron by @AlpinDale in #994
• models: add support for QwenVL by @AlpinDale in #995
• fix: gptq_marlin exception on older GPUs by @AlpinDale in #996
• chore: use ray[adag] dep instead of cuda by @AlpinDale in #997
• quants: improve awq_triton throughput by @AlpinDale in #998
• fix: hermes tool call chat template by @AlpinDale in #999
• core: fix async postprocessor in case of preemption by @AlpinDale in #1000
• vlm: add multi-input support for LLaVA and InternVL models by @AlpinDale in #1002
• tools: fix tool calls to more strictly follow OpenAI format by @AlpinDale in #1003
• fix: LoRA support for Cohere and Jamba models by @AlpinDale in #1004
• spec decode: move ops.advane_step to flash attention backend by @AlpinDale in #1005
• chore: remove peft as a requirement by @AlpinDale in #1006
• chore: keep chunked prefill enabled with prefix caching by @AlpinDale in #1007
• fix: ensure multistep lookahead allocation is compatible with cugraph max capture by @AlpinDale in #1008
• fix: pass APHRODITE_ATTENTION_BACKEND to ray workers by @AlpinDale in #1009
• build: shallow clone cutlass 3.5.1 tag by @AlpinDale in #1010
• chore: skip loading extra bias for qwen2 moe GPTQ by @AlpinDale in #1011
• fix: internvl pipeline parallel by @AlpinDale in #1012
• quants: add support for NVIDIA's ModelOpt checkpoints by @AlpinDale in #1013
• vlm: add support for video modality + llava next video model by @AlpinDale in #1014
• vlm: add support for Qwen2-VL model by @AlpinDale in #1015
• cpu: fix issue with sampling kernels by @AlpinDale in #1016
• cpu: add support for W8A8 quantization via compressed-tensor by @AlpinDale in #1017
• kernel: add meta functions for ops to prevent graph breaks by @AlpinDale in #1019
• chore: move device keys to a constant by @AlpinDale in #1020
• tests: refactor speculative decoding tests to remove the async engine by @AlpinDale in #1021
• vlm: add support for Pixtral model by @AlpinDale in #1022
• core: dump model runner inputs during crash by @AlpinDale in #1023
• chore: remove engine_use_ray by @AlpinDale in #1024
• api: fix logic for deciding if tool parser is used by @AlpinDale in #1025
• quants: add bitsandbytes support for gemma2 model by @AlpinDale in #1026
• cpu: raise error if using encoder-decoder models by @AlpinDale in #1027
• chore: use RoPE cache for MRoPE method by @AlpinDale in #1028
• torch.compile: hide slicing under custom op for inductor by @AlpinDale in #1029
• vlm: fix internvl2 inference with various num_patches by @AlpinDale in #1030
• vlm: support multiple images for qwen-vl by @AlpinDale in #1031
• fix: lazy init _copy_stream by @AlpinDale in #1032
• multi-step: add support for flashinfer attention backend by @AlpinDale in #1033
• api: add sampling/engine option to return only deltas or final output by @AlpinDale in #1035
• fix: multi-step + flashinfer with cuda graphs by @AlpinDale in #1036
• fix: disable chunked prefill and prefix caching for multimodal models by @AlpinDale in #1037
• fix: grouped_topk return type by @AlpinDale in #1038
• core: factor out input preprocessing into a separate class by @AlpinDale in #1039
• fix: skip loading extra bias for Qwen2-VL GPTQ by @AlpinDale in #1040
• torch.compile: allow adding custom compile backends via plugins by @AlpinDale in #1041
• xpu: bump IPEX to 2.3, support GQA by @AlpinDale in #1042
• rocm: add custom paged attention kernels for ROCm by @AlpinDale in #1043
• model: add support for MiniCPM-3 by @AlpinDale in #1044
• torch.compile: fix functionalization by @AlpinDale in #1045
• tpu: implement multi-step scheduling by @AlpinDale in #1046
• fix: clean shutdown issues by @AlpinDale in #1047
• kernel: asymmetric AQ AZP quantization kernels by @AlpinDale in #1048
• chore: remove dead code from triton sampling kernels by @AlpinDale in #1049
• torch.compile: register all-reduce operations as custom ops by @AlpinDale in #1050
• core: add cuda graph support for encoder-decoder models by @AlpinDale in #1051
• quant: add tensor parallel support for bitsandbytes by @AlpinDale in #1052
• api: add mistral function calling format to all models loaded with "mistral" format by @AlpinDale in #1053
• fix: granite logit scale in logit computation by @AlpinDale in #1054
• mamba: enable continuous batching for mamba kernels by @AlpinDale in #1055
• build: fix compilation for causal_conv1d_fwd kernel signature by @AlpinDale in #1057
• fix: add missing logit index increment in sampling metadata prep by @AlpinDale in #1059
• feat: introduce MQAphroditeEngine by @AlpinDale in #1056
• core: support prompt logprobs in multi-step by @AlpinDale in #1060
• distributed: bind only to 127.0.0.1 for local-only usage by @AlpinDale in #1061
• rocm: add more quants, fix _scaled_mm call by @AlpinDale in #1062
• api: non-zero exit code if MQ engine startup fails by @AlpinDale in #1063
• fix: encoder-decoder models for beam search by @AlpinDale in #1064
• api: enable MQAphroditeEngine for embedding models by @AlpinDale in #1065
• rocm: add support for FP8 KV cache in the custom paged attention kernels by @AlpinDale in #1066
• sampler: simplify logits resort in _apply_top_k_top_p by @AlpinDale in #1067
• build: guard against changes in cuda library name by @AlpinDale in #1068
• vlm: add support for molmo vision model by @AlpinDale in #1069
• fix: Phi3.5 Mini and MoE LoRA inference by @AlpinDale in #1070
• rocm: enable multi-step scheduling for rocm by @AlpinDale in #1071
• api: support LoRA lineage and base model metadata management by @AlpinDale in #1072
• fix: validate n in the sampling params by @AlpinDale in #1075
• fix: unexpected kwarg for the legacy API server by @AlpinDale in #1076
• tests: refactor model tests by @AlpinDale in #1078
• vlm: fix feature size calculation for llava-next models by @AlpinDale in #1079
• core: rename PromptInputs,inputs -> PromptType,prompt by @AlpinDale in #1080
• chore: check for torch 2.4.0 when registering custom op by @AlpinDale in #1081
• readme: update attributions by @AlpinDale in #1082
• core: factor out common code in SequenceData by @AlpinDale in #1083
• api: fix admin key being required for authentication by @AlpinDale in #1091
• build: build flash attention kernels inside aphrodite by @AlpinDale in #1085
• VLM: use SequenceData.from_token_counts to create dummy data by @AlpinDale in #1093
• Triton: remove atomic add op from awq triton by @AlpinDale in #1094
• moe: refactor DBRX experts to support FusedMoE by @AlpinDale in #1095
• VLM: refactor composite weight loading logic by @AlpinDale in #1097
• VLM: refactor blip models to support composite weight loading by @AlpinDale in #1098
• misc: rename CudaMemoryProfiler to DeviceMemoryProfiler by @AlpinDale in #1099
• VLM: add support for LLaVA-Onevision model by @AlpinDale in #1100
• spec decode: remove dead code from draft bonus tokens by @AlpinDale in #1101
• VLM: support passing multimodal processor kwargs by @AlpinDale in #1102
• VLM: add pipeline parallelism support for Qwen2-VL by @AlpinDale in #1103
• LoRA: add assertions for SGMV kernels to avoid incorrect results by @AlpinDale in #1104
• distributed: allow IPv6 in APHRODITE_HOST_IP with ZMQ by @AlpinDale in #1105
• Spec Decoding: fix typical acceptance sampler with correct recovered tok IDs by @AlpinDale in #1106
• samplers: improved DRY performance by @AlpinDale in #1108
• LoRA: skip loading unsupported weight modules by @AlpinDale in #1109
• fix: correct FP8 support check on Ada+ GPUs by using compressed-tensors by @AlpinDale in #1110
• CI: bump aphrodite-engine to v0.6.6 by @AlpinDale in #1111

Full Changelog: v0.6.5...v0.6.6