Read the book online: https://amitxshukla.github.io/LLM/
Most engineers today only fine-tune models that other people built. This book goes one level deeper.
You start by writing a tiny transformer by hand, line by line. You break it on purpose, so you learn what each part does. Then you build a proper tokenizer for Sanskrit, collect and clean a real corpus, and adapt a large open model into a genuine Sanskrit specialist that runs on your own machine.
The whole book is written in simple English. Every technical word is explained the first time it appears. There is a glossary at the end.
You know some Python. You have touched PyTorch, or you are willing to learn it as you go. You do not need a PhD, and you do not need a GPU cluster.
Sanskrit teaches you structure and scarcity: sandhi, long compound words, and the hard fact that there is not much clean text. Urdu teaches you mess and script: right-to-left writing, unwritten short vowels, and half the internet's Urdu written in English letters.
Together they cover almost every problem you will meet in a real low-resource language project.
💡 Many chapters now end with a 🧑💻 Runnable code section containing tested, copy-paste code, colored callouts, and diagrams. The full code lives in
code/; see the table below the chapter list.
| Step | Chapter | Status |
|---|---|---|
| 0 | Get your workspace ready | Draft |
| 1 | Build a tiny Sanskrit transformer | Draft |
| 2 | Break your model on purpose | Draft |
| Step | Chapter | Status |
|---|---|---|
| 3 | Understand tokenizers | Draft |
| 4 | Build a Sanskrit tokenizer | Draft |
| 5 | Build an Urdu tokenizer | Draft |
| 6 | Collect your data | Draft |
| 7 | Clean your data | Draft |
| Step | Chapter | Status |
|---|---|---|
| 8 | Rebuild with the modern design | Draft |
| 9 | Run a real training job | Draft |
| 10 | Test it honestly | Draft |
| Step | Chapter | Status |
|---|---|---|
| 11 | Adapt a large open model | Draft |
| 12 | Teach it to follow instructions | Draft |
| 13 | Teach it what a good answer looks like | Draft |
| 14 | Teach it to reason | Draft |
| 15 | Panini: add the rules back in | Draft |
| Step | Chapter | Status |
|---|---|---|
| 16 | Mixture of Experts | Outline |
| 17 | Distillation | Outline |
| 18 | Make it small and serve it | Draft |
| 19 | Long text and RAG | Outline |
| Step | Chapter | Status |
|---|---|---|
| 20 | Speech | Outline |
| 21 | Images and manuscript OCR | Outline |
| 22 | Video | Outline |
| 23 | Agents and tools | Outline |
| 24 | Medical images and heart sounds | Outline |
| Step | Chapter | Status |
|---|---|---|
| 25 | Release your model responsibly | Draft |
- Glossary
- Hardware notes (including DGX Spark)
- Where to find Sanskrit and Urdu text
- Who else is working on this
- Should you build from scratch at all?
- Things worth reading
Every core step now ships with tested, runnable code in code/.
Each folder is self-contained with its own README (what it does, how to run,
hardware, and time).
| Folder | What it runs | Used in |
|---|---|---|
🧠 code/step-01-tiny-transformer/ |
A ~250-line GPT from scratch that babbles Sanskrit | Steps 1–2, 9 |
🔤 code/step-04-sanskrit-tokenizer/ |
Code-point vs. grapheme (akshara) tokenizers | Step 4 |
🧹 code/step-06-data-audit/ |
PDFs → clean corpus, with an OCR health check | Steps 6–7 |
🚀 code/step-11-adapt-base-model/ |
LoRA/QLoRA fine-tuning: dataset → train → chat | Steps 11–13 |
🧩 code/step-14-reasoning/ |
Verifiable rewards + GRPO, and a distillation set | Steps 14, 17 |
❤️ code/step-24-medical-ecg/ |
1D-CNN heartbeat → arrhythmia, imbalance-aware | Step 24 |
🦴 code/step-24-medical-xray/ |
X-ray → fracture by transfer learning | Steps 21, 24 |
🎥 code/step-22-video/ |
Per-frame CNN + temporal Transformer for behaviour | Step 22 |
# quickest taste — see the Sanskrit tokenization "gotcha" in 5 seconds
pip install regex && python code/step-04-sanskrit-tokenizer/devanagari_tokenizer.pyLonger-form notes, a full course PDF, and diagrams live in docs/ and
are indexed in docs/references.md.
| Resource | Format | What it is |
|---|---|---|
| 📘 Fine-Tuning Foundation Models | PDF (58 pp) | Full companion course: language, healthcare multimodal (ECG/X-ray/video), reasoning, private NVIDIA deployment, and an interview Q&A bank |
| 🧠 Tiny-transformer teaching notes | Markdown | Every concept in the from-scratch model, mapped to the code |
| ✍️ Blog: I built a tiny Sanskrit GPT | Markdown | Publish-ready narrative of the tokenization gotcha |
| 🚀 Fine-tuning teaching notes | Markdown | LoRA/QLoRA, SFT, DPO, and the road to reasoning |
| 🗺️ Fine-tuning architecture | Markdown (Mermaid) | The full pipeline + LoRA concept + roadmap diagrams |
| 🖥️ GPU primer (CuTile) | Markdown | Beginner explainer of GPU programming |
The book is written in MyST Markdown. You need Node.js 20 or newer.
# install the MyST command line tool, once
npm install -g mystmd
# start a live preview at http://localhost:3000
myst start
# build the static site into ./_build/html
myst build --htmlTo build a PDF you also need a LaTeX installation:
myst build --pdfEvery push to main triggers the deploy.yml workflow. It builds the site with MyST and publishes it to GitHub Pages.
To turn it on the first time:
- Go to Settings → Pages in this repository.
- Under Source, choose GitHub Actions.
- Push to
main.
.
├── book/ # the chapters, one Markdown file per step
│ ├── intro.md # the long introduction
│ ├── appendix/ # glossary, hardware, corpora, reading list
│ └── images/ # figures used in chapters
├── docs/ # reference PDFs, papers, and notes (not built into the book)
├── code/ # runnable example code for each chapter
├── myst.yml # book config and table of contents
└── .github/workflows/deploy.yml
Adding a new chapter: create the Markdown file in book/, then add one line for it in the toc: section of myst.yml.
Corrections and additions are welcome. The most useful contributions are:
- Tokenizer fertility measurements for languages not covered here
- Evaluation sets written by native speakers
- Reports of what failed for you and why
Please read CONTRIBUTING.md and open an issue before sending a large pull request.
Amit Shukla
- Website: https://AmitXShukla.github.io/LLM/
- GitHub: @AmitXShukla
- YouTube: https://youtube.com/@amit.shukla
- Contact: X.com/@ashuklax
If this book helped you, a star on the repository is genuinely appreciated.
See CITATION.cff, or use:
Amit Shukla. Build Your Own Language Model: A step-by-step book for engineers, using Sanskrit and Urdu. 2026. https://github.com/AmitXShukla/llm
Two licences, because a book is two things at once:
- The writing (everything in
book/anddocs/) is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You may share and adapt it, including commercially, as long as you give credit. - The code (everything in
code/, and all code samples inside chapters) is licensed under the MIT Licence. Use it freely.
Reference PDFs placed in docs/ remain under the licence of their original authors. Do not commit anything you do not have the right to redistribute.