Files

02_bonus_bytepair-encoder

Fix BPE bonus materials (#561 )

Mar 8, 2025

f63f04d · Mar 8, 2025

Name		Name	Last commit message	Last commit date
parent directory ..
gpt2_model		gpt2_model	Set up basic test gh worklows (#79 )	Mar 18, 2024
README.md		README.md	add readme files	Oct 25, 2023
bpe_openai_gpt2.py		bpe_openai_gpt2.py	Ch05 supplementary code (#81 )	Mar 19, 2024
compare-bpe-tiktoken.ipynb		compare-bpe-tiktoken.ipynb	Fix BPE bonus materials (#561 )	Mar 8, 2025
requirements-extra.txt		requirements-extra.txt	add requirements file for bonus notebook	Mar 2, 2024

README.md

Chapter 2: Working with Text Data

compare-bpe-tiktoken.ipynb benchmarks various byte pair encoding implementations
bpe_openai_gpt2.py is the original bytepair encoder code used by OpenAI