Skip to content

Releases: guqiong96/Lvllmds4-x

Release list

lvllmds4-x-v2.3.9

lvllmds4-x-v2.3.9 Latest
Latest

Choose a tag to compare

View all tags
@guqiong96 guqiong96 released this 28 Jul 14:13
lvllmds4-x-v2.3.9
9cdbcef

NOTE: chore: upgrade lk_moe to v2.3.2 for improved DSv4 accuracy

Assets 3
Loading

lvllmds4-x-v2.3.8

lvllmds4-x-v2.3.8

Choose a tag to compare

View all tags
@guqiong96 guqiong96 released this 20 Jul 05:33
lvllmds4-x-v2.3.8
3cfabc7

Inference command for DeepSeek-V4-Flash-DSpark (supports SM80+ architectures, memory-optimized)
NOTE: Reduced --gpu-memory-utilization to 0.85 to reserve space for Triton compilation.

Installation Steps

1. Install CUDA 13.2.1

# Uninstall old CUDA and NVIDIA driver
sudo /usr/local/cuda/bin/cuda-uninstaller   
sudo nvidia-uninstall

# Download and install CUDA 13.2.1
wget https://developer.download.nvidia.com/compute/cuda/13.2.1/local_installers/cuda_13.2.1_595.58.03_linux.run
sudo sh cuda_13.2.1_595.58.03_linux.run

2. Create Python Environment

conda create -n lvllmds4-x python==3.12.11
conda activate lvllmds4-x

# Upgrade libstdcxx-ng (to avoid glibcxx version issues)
conda install -c conda-forge libstdcxx-ng
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Install NUMA library
sudo apt-get install libnuma-dev      # Ubuntu
sudo dnf install numactl-devel        # Rocky Linux

3. Install LvLLM

pip install https://github.com/guqiong96/Lvllmds4-x/releases/download/lvllmds4-x-v2.3.8/lvllmds4_x-2.3.8-cp312-cp312-manylinux_2_34_x86_64.whl

4. Run on dual NVIDIA GeForce RTX 3090 GPUs (24GB VRAM each).


CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=0,3 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
FLASHINFER_DISABLE_VERSION_CHECK=1 \
LVLLM_MOE_NUMA_ENABLED=1 \
LVLLM_ENABLE_MOE_LAYERWISEISE_LOAD=1 \
LK_THREADS=44 \
OMP_NUM_THREADS=44 \
LK_THREAD_BINDING=CPU_CORE \
LVLLM_GPU_PREFETCH_WINDOW=1 \
LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=128 \
LK_POWER_SAVING=1 \
FLASHINFER_DISABLE_VERSION_CHECK=1 \
vllm serve /home/guqiong/Models/DeepSeek-V4-Flash-DSpark \
  --host 0.0.0.0 \
  --port 8070 \
  --tensor-parallel-size 2 \
  --max-model-len 22000 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --served-model-name DeepSeek-V4-Flash-DSpark \
  --compilation_config.cudagraph_mode FULL_DECODE_ONLY \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --dtype bfloat16 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"draft_sample_method":"greedy"}'

RTX Pro 6000 Optimization

Parameter 3090 ×2 RTX Pro 6000 Benefit
CUDA_VISIBLE_DEVICES 0,3 0 Single GPU
--tensor-parallel-size 2 1 No TP needed
--max-model-len 22000 128000 Extended context
--max-num-batched-tokens 8192 16384 Faster Prefill
--gpu-memory-utilization 0.85 0.95 More VRAM available
LVLLM_GPU_RESIDENT_MOE_LAYERS - "0-20" Faster Prefill & Decode

For detailed optimization parameters, see: https://github.com/guqiong96/Lvllm#optimization

Loading