Repository navigation
Reproduced ChatGPT2 (124M): pretrained model + SFT + RLHF !!! #113
jacoboromerodiaz
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
I just finished a project where I built GPT-2 following Karpathy's tutorial and took it all the way to a full instruct model: pretraining, SFT, and RLHF. My goal wasn't to beat any benchmark or other comments in the discussion. It was to understand every part of LLM end-to-end development.
A pretrained base just predicts the next token on raw text, it doesn't follow instructions, doesn't know what an assistant turn looks like, and has no notion of a "preferred" output. So, I continued the pipeline by implementing:
🔧 Stage 1: Supervised Finetuning
🔧 Stage 2: Reinforcement Learning from Human Feedback
These stages are explained in detail in my repo.
🛠️ Other extensions over the baseline
torch.compileactive and cut eval time💰 TOTAL COST: ~$10 (RTX A5000)
📈 HellaSwag across stages
Qualitative outputs are in
finetune/playground.ipynb.Watching the model progress from raw next-token prediction to assistant format, to preference-aligned outputs was super interesting. If anyone needs help to reproduce my models or discuss any potential improvements/fixes, feel free to reach out!
Repo: https://github.com/jacoboromerodiaz/gpt-2
All reactions