Skip to content

v0.9.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 24 Aug 15:33
v0.9.0
c8b1ea6

v0.9.0

It has been a long time. There have been many, many changes between this release and v0.6.7. I'll try to summarize the most important ones, but I'll likely miss quite a lot.

New Models

Transformers backend

You can now load any unsupported model using the integrated transformers backend. By default, if an unsupported model is loaded, Aphrodite will attempt to load it using Transformers if a native implementation doesn't exist.

Quantization Methods

There have been a few quant methods added.

  • NVFP4 - the new datatype supported by Blackwell GPUs. Will also work on Ampere and Hopper using Marlin kernels.
  • MXFP4 - popularized by GPT-Oss. Natively supported by Blackwell; Hopper and Ampere will use Marlin.
  • GPTQAllSpark - optimization for GPTQ models, supported when the model has group_size=-1 and act_desc=False. Seems to provide better perf than Marlin. Enable with -q gptq_allspark.
  • BitBLAS - support for BitNet-quantized 1.58bit models. Will also support GPTQ.
  • TorchAO - support for models quantized using TorchAO.

New Features

There's a lot of new features! Here are some of the more important ones:

  • Sophisticated support for torch.compile
  • DeepGEMM for DeepSeek-V3
  • Expert Load Balancing and Expert Parallel
  • FlexAttention
  • TreeAttention
  • Differential Flash Attention
  • Dual Chunk Flash Attention
  • Flash Attention V3 (supported only for A100/H100 GPUs)
  • Disaggregated Prefill – run separate instances of Aphrodite for prefill and decode. Boosts throughput by eliminating compute starvation caused by expensive prefill requests at the cost of requiring more GPUs.
  • Async Scheduling via NanoFlow – provides a ~13% throughput and latency boost (--async-scheduling)
  • Async Tensor Parallel – provides an ~8% throughput and latency boost on Hopper GPUs, ~3% on Ampere (-O '{"level":3, "compile_sizes": [512], "pass_config": {"enable_async_tp": true}}')
  • Mirostat Sampling – it returns!
  • String/Phrase Banning – also known as "anti-slop sampler", this one allows you to specify a list of phrases to ban from generation. Pass banned_strings as a parameter in the API request body, and provide it with a list of strings.

As always, many thanks to the vLLM!

What's Changed

New Contributors

Full Changelog: v0.6.7...v0.9.0

Contributors

dsk7, Nero10578, and 4 other contributors
Assets 3
Loading