lvllmds4-v2.3.9

NOTE: chore: upgrade lk_moe to v2.3.2 for improved DSv4 accuracy

Command for 2× RTX 5060 Ti

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=1,2 \
NCCL_SOCKET_IFNAME=lo \
NCCL_IB_DISABLE=1 \
GLOO_SOCKET_IFNAME=lo \
NCCL_SOCKET_TIMEOUT=600000 \
LVLLM_MOE_NUMA_ENABLED=1 \
LK_THREADS=44 \
OMP_NUM_THREADS=44 \
LK_THREAD_BINDING=CPU_CORE \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=128 \
LK_POWER_SAVING=1 \
FLASHINFER_DISABLE_VERSION_CHECK=1 \
vllm serve /home/guqiong/Models/DeepSeek-V4-Flash-DSpark \
  --host 0.0.0.0 \
  --port 8070 \
  --tensor-parallel-size 2 \
  --max-model-len 32000 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --served-model-name DeepSeek-V4-Flash-DSpark \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "mode": "VLLM_COMPILE"}' \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096 \
  --dtype bfloat16 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

RTX Pro 6000 Optimization

Parameter	5060 Ti ×2	RTX Pro 6000	Benefit
CUDA_VISIBLE_DEVICES	1,2	0	Single GPU
--tensor-parallel-size	2	1	No TP needed
--max-model-len	32000	128000	Extended context
--max-num-batched-tokens	4096	16384	Faster Prefill
LVLLM_GPU_RESIDENT_MOE_LAYERS	-	"0-20"	Faster Prefill & Decode

For detailed optimization parameters, see: https://github.com/guqiong96/Lvllm#optimization

lvllmds4-v2.3.7

fix: apply vllm PR #47586 to resolve AttributeError: .../tilelang/lib/libcudart_stub.so: undefined symbol: cudaDeviceReset

feat: lk_moe==2.3.1 add FP16 activation support for MOE V2
feat: lk_moe==2.3.1 support dynamic FP8 quantization granularity (groupK/groupN), for example : deepreinforce-ai/Ornith-1.0-35B-FP8
feat: lk_moe==2.3.1 add activation_type config for swigluoai/relu2/silu, for example: nvidia/MiniMax-M3-NVFP4

lvllmds4-v2.3.3

perf: uodate lk_moe to v2.2.0, optimize avx avx512 kernel for ~10% inference speedup

lvllmds4-v2.3.2

fix bug: error when run awq4bit nvfp4 models

lvllmds4-v2.3.1

Added support for the deepseek-ai/DeepSeek-V4-Flash-DSpark model.
DSpark speculative decoding :
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

lvllmds4-v2.3.0

Optimize GPU prefill speed, CPU AVX512 optimization, removed LVLLM_GPU_RESIDENT_MOE_EXPERTS

lvllmds4-v2.2.1

Installation Steps

1. Install CUDA 13.2.1

# Uninstall old CUDA and NVIDIA driver
sudo /usr/local/cuda/bin/cuda-uninstaller   
sudo nvidia-uninstall

# Download and install CUDA 13.2.1
wget https://developer.download.nvidia.com/compute/cuda/13.2.1/local_installers/cuda_13.2.1_595.58.03_linux.run
sudo sh cuda_13.2.1_595.58.03_linux.run

2. Create Python Environment

conda create -n lvllmds4 python==3.12.11
conda activate lvllmds4

# Upgrade libstdcxx-ng (to avoid glibcxx version issues)
conda install -c conda-forge libstdcxx-ng
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Install NUMA library
sudo apt-get install libnuma-dev      # Ubuntu
sudo dnf install numactl-devel        # Rocky Linux

3. Install LvLLM

pip install https://github.com/guqiong96/Lvllmds4/releases/download/lvllmds4-v2.2.1/lvllmds4-2.2.1-cp312-cp312-manylinux_2_34_x86_64.whl

4. Run

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
LVLLM_MOE_NUMA_ENABLED=1 \
LVLLM_ENABLE_MOE_LAYERWISEISE_LOAD=1 \
LK_THREADS=44 \
OMP_NUM_THREADS=44 \
LK_THREAD_BINDING=CPU_CORE \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_RESIDENT_MOE_LAYERS=0-1 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=512 \
LVLLM_ENABLE_NUMA_INTERLEAVE=1 \
FLASHINFER_DISABLE_VERSION_CHECK=1 \
vllm serve \
    --model ~/Models/DeepSeek-V4-Flash \
    --host 0.0.0.0 \
    --port 8070 \
    --tensor-parallel-size 2 \
    --max-model-len 20000 \
    --gpu-memory-utilization 0.95 \
    --trust-remote-code \
    --served-model-name DeepSeek-V4-Flash \
    --compilation_config.cudagraph_mode FULL_DECODE_ONLY \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-num-batched-tokens 8192 \
    --dtype bfloat16 \
    --max-num-seqs 4 \
    --compilation_config.mode VLLM_COMPILE \
    --enable-auto-tool-choice \
    --kv-cache-dtype fp8 \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --reasoning-parser deepseek_v4