Repository navigation
Modular Reimplementation #114
aleksandr-agadzhanov
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I had a go at reimplementing this repository from scratch using the Let's reproduce GPT-2 (124M) YouTube tutorial and here is what I came up with:
https://github.com/aleksandr-agadzhanov/nanogpt-reimplementation
Here I trained a 98M parameter model for one epoch over FineWeb-Edu 10BT. The run took approximately 1.5 hours on a GPU Box from Lambda Cloud with 8 x A100 40GB SXM4 GPUs. The total cost was approximately $40, including transferring the training data to the instance.
The final recorded metrics were:
These results put this one-epoch run ahead of the cited GPT-2 124M comparison values
on all three reported metrics. It did not outperform the cited GPT-3 124M result in
this experiment. However, the learning trend suggested that additional epochs could
eventually close the gap and potentially surpass GPT-3 124M.
I also used my custom tokenizer which I implemented as part of this repo:
https://github.com/aleksandr-agadzhanov/minbpe-reimplementation
Hope it is useful to someone learning LLMs from first principles and following the YouTube guide like I did.
All reactions