conda create --name lavida python=3.13
conda activate lavida
pip install -e .[lavida]
pip install wheel
MAX_JOBS=32 pip install flash-attn==2.7.4.post1 --no-build-isolation
pip install jupyter notebook
pip install -U huggingface_hub[hf_xet] --force-reinstall
Please download checkpoints from [Huggingface]
Example inference script is provided as demo_sparse_lavida.py. This script will run both the standard decoding and sparse decoding with token truncation, and benchmark output latency.
See Training Readme for general training of LaViDa-O.
Please run the training script to finetune the model for sparse parameterization. The script is just standard SFT script with the following key flags for sparsity:
--block_causal True \
--gen_enc_add_pos_emb True \
--num_register_tokens 64,64 \
--num_register_groups 25,25 \
Both the model and code are licensed with Adobe Research License, which is included here [License.pdf].
