Lil' Bobby - Compressing RoBERTa_large to small networks

This repository hosts the source code for our master's thesis in Computer Science at IT-University, Copenhagen in spring 2021.

This thesis explores the effects of knowledge distillation, pruning, and quantization of the RoBERTa language model on SST-2, QQP, and MNLI. The goal is to learn the extent to which a finetuned RoBERTa language model can be compressed while providing tolerable performance.

Name		Name	Last commit message	Last commit date
Latest commit History 763 Commits
analysis		analysis
common		common
compression		compression
custom		custom
embedding		embedding
misc		misc
preprocessing		preprocessing
.gitignore		.gitignore
README.md		README.md
Thesis.pdf		Thesis.pdf
__init__.py		__init__.py
analyze.py		analyze.py
compress.py		compress.py
dependencies.txt		dependencies.txt
evaluate.py		evaluate.py
experiment.py		experiment.py
finetune.py		finetune.py
format_results.py		format_results.py
interactive.py		interactive.py
preprocess.py		preprocess.py
run_all_seeds.py		run_all_seeds.py
run_distill.py		run_distill.py
run_extra_compression.py		run_extra_compression.py
run_pt_compression.py		run_pt_compression.py
seeds.txt		seeds.txt
task_difficulty.py		task_difficulty.py
task_word_difficulty.py		task_word_difficulty.py
validate_augmented.py		validate_augmented.py

pocketML/lil_bobby

Folders and files

Latest commit

History

Repository files navigation

Lil' Bobby - Compressing RoBERTa_large to small networks

About

Resources

Stars

Watchers

Forks

Languages