I like building things to learn how they work, as I've been learning language models been building components to develop intuition.
Not sure will continue working on this repo but I hope I build this up into a full nanogpt repo, not sure specifically about cuda stuff and optimizer stuff everything else I think I know how to vaugely do and can get there.
Would highly recommend taking an indepth tutorial and writing it in another language (ideally one you don't know very well)
"tutorial" for backprop and gradient descent
