Critical Data Size of Language Models from a Grokking Perspective

Prerequisites

Execute the following command to re-produce our results:

sh run_grokking_on_imdb.sh

sh run_grokking_on_yelp.sh

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
datasets		datasets
figures		figures
grokking		grokking
model-wise_grokking		model-wise_grokking
sh_command		sh_command
slurm		slurm
training_config		training_config
utils		utils
.gitignore		.gitignore
README.md		README.md
eval.py		eval.py
generate_gitignore.py		generate_gitignore.py
grokking.slurm		grokking.slurm
grokking_imdb.slurm		grokking_imdb.slurm
grokking_yelp.slurm		grokking_yelp.slurm
run_classifier.py		run_classifier.py
run_grokking_on_gpt_with_modular.sh		run_grokking_on_gpt_with_modular.sh
run_grokking_on_gpt_with_wikitext.slurm		run_grokking_on_gpt_with_wikitext.slurm
run_grokking_on_imdb.sh		run_grokking_on_imdb.sh
run_grokking_on_instrution_tuning.sh		run_grokking_on_instrution_tuning.sh
run_grokking_on_yelp.sh		run_grokking_on_yelp.sh
test.py		test.py
train_language_modeling.py		train_language_modeling.py
training_reference_models.py		training_reference_models.py