Repository navigation
Re-building and improving GPT-2 #106
garg-aayush
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Over the last couple of weeks, I followed @karpathy’s ‘Let’s Reproduce GPT-2’ video religiously—making notes, implementing the logic line by line, and completing a re-implementation of GPT-2 from scratch.
I went a few steps further by implementing some of the improvements suggested by @karpathy (such as learning rate adjustments and data loader fixes), along with modern enhancements like RoPE and SwiGLU-FFN. My best-performing experiment
gpt2-rope, achieved a validation loss of 2.987 and a HellaSwag accuracy of 0.320.I really loved the whole process of writing the code, running multiple trainings and gradually seeing the losses improve. I learnt so much about LLMs pre-training from this single video. Honestly, the $200 I spent on compute over these two weeks was the best money I’ve spent lately. Learned a ton and had fun.
I have made sure to log everything, the code, training runs, checkpoints, notes:
All reactions