Skipping refactor #234

…t#4376) Co-authored-by: WoosukKwon <woosuk.kwon@berkeley.edu>

Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>

…fill (vllm-project#4309)

…oject#4373)

Co-authored-by: Simon Mo <simon.mo@hey.com>

…t#4348)

…les (vllm-project#4343) Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…4355)

…m-project#4406)

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>

…errupted (vllm-project#4363)

) Co-authored-by: DefTruth <31974251+deftruth@users.noreply.github.com>

Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com> Co-authored-by: Travis Johnson <tjohnson31415@gmail.com>

…ect#4402)

…ct#4418)

Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com> Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>

…project#3922) Co-authored-by: alexm <alexm@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

…project#4444)

…ect#4165) Co-authored-by: Lei Wen <wenlei03@qiyi.com>

Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com> Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Philipp Moritz <pcmoritz@gmail.com> Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Co-authored-by: mgoin <michael@neuralmagic.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>

…int (vllm-project#3467) Co-authored-by: Lily Liu <lilyliupku@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…ersion (vllm-project#4467)

…vllm-project#4494) Co-authored-by: Simon Mo <simon.mo@hey.com>

… obtain the CUDA version. (vllm-project#4173) Signed-off-by: AnyISalIn <anyisalin@gmail.com>

…ect#4503)

…-project#4534) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>

…#4237) Co-authored-by: Lei Wen <wenlei03@qiyi.com>

…ect#4142) Co-authored-by: Lei Wen <wenlei03@qiyi.com> Co-authored-by: Sage Moore <sagemoore@utexas.edu>

…vllm-project#4357)

This PR updates the tuning script for the fused_moe kernel to support FP8 and also adds configurations for TP4. Note that for the configuration I removed num_warps and num_stages for small batch sizes since that improved performance and brought the benchmarks on par with the numbers before in that regime to make sure this is a strict improvement over the status quo. All the numbers below are for mistralai/Mixtral-8x7B-Instruct-v0.1, 1000 input and 50 output tokens. Before this PR (with static activation scaling): qps = 1: 9.8 ms ITL, 0.49s e2e latency qps = 2: 9.7 ms ITL, 0.49s e2e latency qps = 4: 10.1 ms ITL, 0.52s e2e latency qps = 6: 11.9 ms ITL, 0.59s e2e latency qps = 8: 14.0 ms ITL, 0.70s e2e latency qps = 10: 15.7 ms ITL, 0.79s e2e latency After this PR (with static activation scaling): qps = 1: 9.8 ms ITL, 0.49s e2e latency qps = 2: 9.7 ms ITL, 0.49s e2e latency qps = 4: 10.2 ms ITL, 0.53s e2e latency qps = 6: 11.9 ms ITL, 0.59s e2e latency qps = 8: 11.9 ms ITL, 0.59s e2e latency qps = 10: 12.1 ms ITL, 0.61s e2e latency

Remove the device="cuda" declarations in mixtral as promised in vllm-project#4343

…to be provided (vllm-project#4273)

…n is not 1 and max_tokens is large & Add tests for preemption (vllm-project#4451)

…vllm-project#4524)

Co-authored-by: Cade Daniel <edacih@gmail.com>

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

…t#4552)

…lm-project#4559)

…ject#4405) Co-authored-by: SangBin Cho <rkooo567@gmail.com>

…4267) Co-authored-by: simon-mo <simon.mo@hey.com>

…ject#4566)

vllm-project#4567)

…m-project#4518)

…default to None (vllm-project#4586)

Co-authored-by: LiuXiaoxuanPKU <llilyliupku@gmail.com>

… Dynamic/Static Activations) (vllm-project#4527) Follow on to vllm-project#4332 to enable FP8 checkpoint loading for Mixtral and supersedes vllm-project#4436. This PR enables the following checkpoint loading features for Mixtral: Supports loading fp8 checkpoints for Mixtral, such as this "nm-testing/Mixtral-8x7B-Instruct-v0.1-FP8" test model Supports static or dynamic activation quantization with static weight quantization (all per tensor) Supports different scales for each expert weight Supports Fp8 in QKV layer Notes: The Expert Gate/Router always runs at half / full precision for now. If there are different weight scales between QKV layer (for separate QKV weights), they are re-quantized using layer.weight_scale.max() so we can have a single gemm for performance.

…trics (vllm-project#3937)

Commits on May 7, 2024

stray issues

robertgshaw2-neuralmagic committed May 7, 2024

1 Configuration menu

View commit details

Copy full SHA for 50c1029

Browse repository at this point

Copy the full SHA

50c1029 View commit details

Browse the repository at this point in the history

Commits on May 9, 2024

fixed torch reinit

Robert Shaw committed May 9, 2024

1 Configuration menu

View commit details

Copy full SHA for 0300194

Browse repository at this point

Copy the full SHA

0300194 View commit details

Browse the repository at this point in the history

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skipping refactor #234

Skipping refactor #234

Commits on May 6, 2024

Commits on May 7, 2024

Commits on May 8, 2024

Commits on May 9, 2024

Commits on May 11, 2024

Commits on May 12, 2024

Commits on May 13, 2024