Skip to content

Latest commit

 

History

History
18 lines (14 loc) · 1.4 KB

lightllm_integration.md

File metadata and controls

18 lines (14 loc) · 1.4 KB
Raw

LightLLM Integration

You can use LightLLM as an optimized worker implementation in FastChat. It offers advanced continuous batching and a much higher (~10x) throughput. See the supported models here.

Instructions

  1. Please refer to the Get started to install LightLLM. Or use Pre-built image

  2. When you launch a model worker, replace the normal worker (fastchat.serve.model_worker) with the LightLLM worker (fastchat.serve.lightllm_worker). All other commands such as controller, gradio web server, and OpenAI API server are kept the same. Refer to --max_total_token_num to understand how to calculate the --max_total_token_num argument.

    python3 -m fastchat.serve.lightllm_worker --model-path lmsys/vicuna-7b-v1.5 --tokenizer_mode "auto" --max_total_token_num 154000

    If you what to use quantized weight and kv cache for inference, try

    python3 -m fastchat.serve.lightllm_worker --model-path lmsys/vicuna-7b-v1.5 --tokenizer_mode "auto" --max_total_token_num 154000 --mode triton_int8weight triton_int8kv