This repository contains my implementation and learning notes based on Andrej Karpathy's makemore lecture series.
部分注释与文章内容使用 AI 辅助整理。
makemore takes one text file as input, where each line is assumed to be one training thing, and generates more things like it. Under the hood, it is an autoregressive character-level language model, with a wide choice of models from bigrams all the way to a Transformer (exactly as seen in GPT).
Current implementation follows a few key papers:
- Bigram (one character predicts the next one with a lookup table of counts)
- MLP, following Bengio et al. 2003
- CNN, following DeepMind WaveNet 2016
- RNN, following Mikolov et al. 2010
- LSTM, following Graves et al. 2014
- GRU, following Kyunghyun Cho et al. 2014
- Transformer, following Vaswani et al. 2017
The included names.txt dataset has the most common 32K names from ssa.gov for the year 2018:
emma
olivia
ava
isabella
sophia
charlotte
...
Start training:
$ python makemore.py -i names.txt -o namesTraining progress, logs, and model checkpoints will be saved to names/. The default model is a tiny ~200K parameter Transformer. For more training configurations, see the argparse section in the code.
To sample from the best checkpoint during or after training:
$ python makemore.py -i names.txt -o names --sample-onlyExample generated names:
dontell
khylum
camatena
aeriline
najlah
sherrith
ryel
irmi
taislee
mortaz
akarli
maxfelynn
biolett
zendy
laisa
halliliana
goralynn
brodynn
romima
chiyomin
loghlyn
melichae
mahmed
irot
helicha
besdy
ebokun
lucianno
PyTorch is the only requirement.
MIT