Skip to content

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

1 watching

Forks

Repository files navigation

Building Domain-Specific Sub-Models from Large Language Models using Pruning

Implementation of the following thesis project: Building Domain-Specific Sub-Models from Large Language Models using Pruning.


Acknowledging the benefits of Wanda pruning method, which prunes LLMs based on both weights and input actiations:


Image sourced from https://github.com/locuslab/wanda.git.

The goal of this project is to focus on the impact of calibration data in the pruing process. Specifically understanding whether pruning using domain-specific calibration sampels leads to domain-specific sub-models. Or anyway if it has an impact on the performance of these sub-models on specific domains.

Setup

Installation instructions can be found in INSTALL.md.

Usage

The datasets used have been doenloaded from Hugging Face and are available for free download. To correctly prune your models you should replace the dataset directory with the one where you downloaded the dataset in data.py.

Below is an example command for pruning CodeLlama-7B-Instruct with Wanda, to achieve unstructured 50% sparsity, using openbookqa as calibration set.

python main.py \
    --model codellama/CodeLlama-13b-Instruct-hf \
    --prune_method wanda \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --pruning_dataset openbookqa
    --save out/codellama-7B-inst-pruned_openbookqa/unstructured/wanda/ 

The following is an overview of the arguments:

  • --model: The identifier for the model on the Hugging Face model hub.
  • --cache_dir: Directory for loading or storing LLM weights.
  • --prune_method: wanda
  • --sparsity_ratio: Denotes the percentage of weights to be pruned.
  • --sparsity_type: Specifies the type of sparsity [unstructured, 2:4, 4:8].
  • --pruning_dataset: The dataset to use as calibration set.
  • --save: Specifies the directory where the result will be stored.

For structured N:M sparsity, set the argument --sparsity_type to "2:4" or "4:8".

Evaluation of sub-models

Each sub-model obtained after pruning has been evaluated on four domains: Code Generation, Commonsense Reasoning, Translation (en-fr) and Math. For each of these domains different metrics and benchmarks have been used.

Code Generation

The accuracy of sub-models on code generation tasks is assessed using the pass@k metric on two renowned benchmarks: HumanEval and MBPP. The pass@k metric is able to evaluate the functional correctness of the code generated, meaning that a sample of generated code is considered correct if it passes a set of unit tests. At least k code samples are generated per problem (specifically a number n), and, having the number of correct samples c (the ones that pass all the unit tests), the evaluation score is computed as:

pass@k formula

To run the evaluation for HumanEval you should be in the right folder (HumanEval) and run the following command:

python eval_pal.py \
    --logdir Gemma-Instruct-7b-pruned_apps \
    --language python \
    --dataroot data

The following is an overview of the arguments:

  • --logdir: The directory where your pruned model's weights are stored.
  • --language: The programming language on which the evaluation will be performed (e.g., python, java, javascript, etc.).
  • --dataroot: The folder containing the HumanEval dataset used for evaluation.

To run the evaluation for MBPP you should be in the right folder (Evaluation_MBPP/MBPP) and run the following command:

python eval_pal.py \
    --logdir Gemma-Instruct-7b-pruned_apps \
    --language python \
    --dataroot data

The following is an overview of the arguments:

  • --logdir: The directory where your pruned model's weights are stored.
  • --dataroot: The folder containing the HumanEval dataset used for evaluation.

Commonsense Reasoning

Commonsense Reasoning focuses on the model’s ability to understand and reason about everyday situations and concepts, assessing the model’s capacity to make logical inferences, predict outcomes, and comprehend implicit knowledge based on common sense. The evaluation is made using zero-shot accuracy. Contrary to few-shot, where the model is given a few demonstrations of the task at inference time, zero-shot evaluation involves giving the model a natural language instruction describing the task, with no demonstrations/examples of desired outcomes.

The framework used for zero-shot accuracy is the same used originally in Wanda, and an explanation of how to use it can be found in their README. The datasets used as benchmark in this case are openbookqa, winogrande, boolq, ARC_Easy, ARC_Challenge, hellaswag.

After having installed the appropriate framweork, the evaluation can be started through the following command:

python ZS-evaluation.py \
    --dataset apps \
    --eval-zero-shot \
    --csr

If the model to be evaluated is a foundational (not-pruned) one, instead of dataset use --model and the name of the hugging face model. For pruned models, instead, specify only the pruned dataset, and change the code in ZS-evaluation.py, so that the prefix present in the method get_weight_dir is the correct one for your sub-model.

Translation

To effectively evaluate translation performance, METEOR was used as a metric. METEOR offers enhanced accuracy as it incorporates not only precision, the proportion of the matched n-grams out of the total number of n-grams in the evaluated translation, but also recall. Recall, which is the proportion of the matched n-grams out of the total number of n-grams in the reference translation, is extremely important for assessing the quality of the output, as it reflects to what degree the translation covers the entire content of the translated sentence.

To ensure the accuracy computation is as valuable and reliable as possible, for each test sample, the answer is generated by the model five times. METEOR accuracy is then calculated independently for each of these five translations by comparing them to the reference translation of the test. The highest accuracy among these five translations is retained. The datasets used for these tests are wmt14 and opus_books.

To run the evaluation the following command is needed:

python samples_generation_transl.py \
    --dataset song_translation \
    --eval_dataset wmt14 

If the model to be evaluated is a foundational (not-pruned) one, instead of dataset use --model and the name of the hugging face model. For pruned models, instead, specify only the pruned dataset, and change the code in samples_generation_transl.py, so that the prefix present in the method get_weight_dir is the correct one for your sub-model. eval_dataset.py represents the dataset used for evaluation, the one containing the reference solutions.

Math

The evaluation of models on math tasks is made leveraging the “grade-school-math” dataset, known as “GSM8K”. The accuracy is calculated as the 8-shot accuracy, which involves assessing the model’s performance based on its ability to accurately generate text or make predictions after being exposed to a limited amount of data (8 samples).

To run the evaluation the GSM8K must be downloaded and the following command is needed:

python math_eval.py \
    --model_name_or_path Gemma-Instruct-7b-pruned_math_qa \
    --data_root gsm8k

Where --model_name_or_path is the directory where the model's weights are stored and --data_root is the directory of the downloaded gsm8k dataset.

Results on Gemma-1.1-7b-it

The following is an example of the results obtained for Gemma's sub-models. The accuracy for each domain is calculated as the average accuracy registered across all the benchamarks used for the specific domain.

pruning_dataset Code CSR Translation Math
- ~52.9% ~40.57% ~50.41% ~46.4%
python_alpaca ~46.99% ~38.23% ~45.75% ~32.44%
apps ~47.07% ~36.47% ~45.84% ~32.2%
github_python ~46.47% ~37.34% ~45.23% ~33.6%
openbookqa ~43.23% ~38.9% ~45.63% ~26.9%
hellaswag ~40.82% ~38.4% ~44.28% ~26.99%
commonsenseqa ~41.61% ~38.45% ~45.21% ~27.29%
wmt14 ~38.58% ~38.34% ~48.55% ~27.14%
song_translation ~40.91% ~37.58% ~48.29% ~29.7%
opus_books ~41% ~39% ~48.6% ~26.8%
math_qa ~44.44% ~37.65% ~45.38% ~35.25%
math_instruct ~45.26% ~37.95% ~45.15% ~35.10%
math_orca ~45.05% ~37.39% ~45.6% ~36.16%
c4 ~42.2% ~38.1% ~46.74% ~29%

Comparison of sub-models' parameter masks

An analysis of the internal structure of the obtained domain-specific sub-models has been conducted to understand whether the way models are pruned affects their specificity to certain domains. To plot the jaccard distance of layers between two sub-models you can use the Visualization/compute_similarity.py script, while for the penzai representation of the weight matrices of the sub-models the notebook Visualization/Visualization_llm.ipynb can be used.

The following is an example of the outcomes that can be derived:

vis1
The first two images (left and center) represent the visualization of the weight matrices “q_proj” of layer 12 of two sub-models obtained by pruning Gemma-7B-it respectively with math_orca dataset (left) and math_qa datasets. The last image (right) represents the weights that differ between the two matrices (red dots).

vis2
The first two images (left and center) represent the visualization of the weight matrices “q_proj” of layer 12 of two sub-models obtained by pruning Gemma-7B-it respectively with math_orca dataset (left) and opus_books datasets. The last image (right) represents the weights that differ between the two matrices (red dots).

vis2
Comparison of the pruning masks of different sub-models obtained by pruning Gemma-7B-it with math_qa, math_orca and opus_books.

From the previous images it is clear that same-domain sub-models tend to have higher similarities in terms of retained weights post-pruning with respect to different-domain ones.

Inference speedup

The inference speed of the original and pruned models was calculated using CUTLASS to sparsify the weight matrices. A speedup was registered with the pruned models. This speedup was expected as part of the trade-off of pruning: sacrificing accuracy for computational improvements. To reproduce the results this blog post can be followed.

Speedup results

Model Layer Dense Sparse Speedup
CodeLlama-I-13B q_proj 0.449 0.329 1.365x
k_proj 0.450 0.330 1.363x
v_proj 0.450 0.330 1.361x
o_proj 0.450 0.331 1.360x
up_proj 1.014 0.998 1.016x
gate_proj 1.010 1.000 1.010x
down_proj 1.061 0.881 1.204x

Acknowledgement

This repository is build upon the Wanda repository.

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages