- script usage
This repository contains the official implementation of our ACL 2025 paper "DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts".
Step 1: Create a new conda environment:
conda create -n dive python=3.9
conda activate dive
Step 2: Install relevant packages
conda install pytorch==2.7.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install -r requirements.txt
Put training datasets in the datasets folder.