Baby Llama

Train and run a small Llama 2 model from scratch on the TinyStories dataset.

• Based on karpathy/llama2.c
• Based on eniompw/DisneyGPT

Baby Llama Code Example:

Baby Llama 105 Tokens on Colab

• Iters vs Val Loss Learning Words and Grammar Visualised
• 105 Token Vocab

!cd llama2.c && python tinystories.py train_vocab --vocab_size=256
trainer_interface.cc(558) LOG(INFO) Alphabet size=102
Vocabulary size is smaller than required_chars. 256 vs 361.

More Tokens & Larger Models

Ref:

• Training
• Models
• Pretokenized TinyStories