The Arithmetic of Intelligence

A problem course in modern machine learning, from the pathological valley to direct preference optimization.

By Anshu Avinash.

A textbook that teaches machine learning the way physics is taught to engineers: as a small set of results you derive, drill, and then recognise everywhere. Seventeen chapters, five appendices, fifty core derivations, roughly two hundred exercises with worked solutions, and eight labs whose acceptance criterion is a number your derivation predicted in advance.

Nothing in the book asks you to memorise a benchmark score, a release date, or an author list. Where a constant from the literature is needed it is given, the way g = 9.8 m/s² is given in a physics paper. What is never given is the model you must build to use it.

---

Contents

Part	Chapters
I — Foundations	1 The Mathematical Toolkit · 2 The Objects
II — The Classical Era (2012–2022)	3 The Geometry of Training · 4 The Architecture of Depth · 5 Memory and Gates · 6 Attention · 7 The Economics of Scale · 8 The Machinery of Scale · 9 Measurement · 10 Compression and Occam
III — The Modern Era (2022– )	11 The Price of a Token · 12 Bytes over FLOPs · 13 Sparsity and Thrift · 14 Teaching Preferences · 15 Thinking at Inference Time · 16 Generation by Denoising · 17 The Whole Lifecycle
Appendices	A The Derivation Bank · B The Trap Taxonomy · C The Lab Manual · D Solutions · E Notation and Glossary

Two editions are built from the same source: a linked web edition (src/index.html, math rendered in-browser by KaTeX) and a PDF (math pre-rendered, fonts embedded, fully self-contained). Both are deployed to GitHub Pages on every push to main.

---

Building

Requirements: Node 18+, Python 3.10+, and the system libraries WeasyPrint needs (Pango, cairo, GDK-PixBuf — on Debian/Ubuntu: apt install libpango-1.0-0 libpangoft2-1.0-0 libcairo2 libgdk-pixbuf-2.0-0).

make install     # npm deps + Python venv with pip dependencies
make pdf         # -> dist/The-Arithmetic-of-Intelligence.pdf

Other targets:

make figure      # regenerate the cover figure from its equations
make audit       # run the source consistency checks
make numbers     # recompute the derived figures and check the source agrees
make check       # both of the above
make serve       # preview the web edition at localhost:8000
make clean       # remove build artefacts

How the build works

The web edition is the canonical source. The PDF is derived from it in four steps, because the PDF renderer executes no JavaScript and has no network access:

1. build/coverfig.py → figures/cover-figure.svg Generates the cover figure by actually running the optimisers (see below).

2. build/prerender.js → .cache/rendered/ Walks every \( … \) and \[ … \] expression in src/*.html and replaces it with static KaTeX markup — about 2,300 expressions. Strips the CDN <script> and <link> tags. Fails loudly rather than silently dropping an expression.

3. build/assemble.py → .cache/book-print.html Merges the twenty-two chapter and appendix files into one document with part dividers, a table of contents, and running page numbers. Inlines build/print.css, the KaTeX stylesheet, and all fonts as base64 data URIs, so the output is a single self-contained file.

4. build/topdf.py → dist/ Renders with WeasyPrint (chosen over headless-Chrome for its @page support: real page geometry, running page numbers, and target-counter so the contents page resolves actual page numbers) and stamps PDF metadata.

---

The cover figure

The cover is not decoration. It is a plot of the book's first derivation.

build/coverfig.py sets up the loss L = ½(x² + 100y²) — an ill-conditioned valley with condition number κ = 100 — and then runs two optimisers on it:

• plain gradient descent at 99% of its stability ceiling η < 2/λ_max, which ricochets between the valley walls;
• heavy-ball momentum, which damps that oscillation and accelerates along the floor.

After seventy steps momentum has reached the minimum while gradient descent is still a quarter of its starting distance away. That gap is the √κ speedup of Derivation 3.2, drawn rather than asserted.

One honest note: optimally tuned momentum (η = (2/(√λ_max+√λ_min))², β = ((√κ−1)/(√κ+1))²) overshoots so hard on its first swings that it leaves the frame. The figure uses well-tuned but less aggressive settings (β = 0.90, η = 0.008) that stay in view and still converge an order of magnitude faster. The caption says "gradient descent and momentum on an ill-conditioned valley", which is accurate.

---

Repository layout

.
├── src/                    # the book — canonical source, also the web edition
│   ├── index.html          #   cover, preface, contents
│   ├── book.css            #   screen styles
│   ├── chapter-01..17.html
│   └── appendix-a..e.html
├── build/
│   ├── coverfig.py         # cover figure from its equations
│   ├── prerender.js        # LaTeX -> static KaTeX
│   ├── assemble.py         # merge + inline into one print document
│   ├── print.css           # page geometry, running heads, break rules
│   └── topdf.py            # render + metadata
├── tools/
│   ├── audit.py            # source consistency checks (11 classes)
│   ├── numbers.py          # recompute derived figures and check source
│   └── renumber.py         # chapter-renumber utility (run-once, guarded)
├── figures/
│   └── cover-figure.svg    # generated by build/coverfig.py
├── .github/workflows/
│   └── pages.yml               # build PDF + deploy both editions to GitHub Pages
├── Makefile
├── package.json            # katex, the three typefaces
└── requirements.txt        # weasyprint, pypdf

.cache/ and dist/ are intermediate/output directories and are git-ignored. The PDF is deployed to GitHub Pages alongside the web edition.

---

The audit

make audit runs eleven classes of check over the source. They exist because this book was drafted in batches, and every one of them caught a real error at some point:

1. Structure — every chapter carries the same furniture: lead paragraph, derivation and worked-example boxes, closing section, readings, graded A/B/C exercises, a gate.
2. Links — every internal href resolves.
3. Markup — <div>, display-math delimiter, and <details> pairs balance in every file.
4. Derivation references — every "Derivation N.M" cited in a chapter is actually defined there.
5. Solution coverage — every B and C exercise has a matching solution in Appendix D.
6. Prose — chapter openings are not built from one template. This check exists because an early draft had twelve of sixteen chapters opening with the literal words "This chapter", which is invisible while writing and obvious while reading.
7. Back-references — a chapter that says "Chapter N's X" is asserting that Chapter N contains X.
8. Derivation bank — every chapter is represented in Appendix A, no entry spans several chapters, and no chapter carries more bank entries than it has boxes to teach them in.
9. Lab anchoring — every lab in Appendix C runs alongside the chapter of the derivation its acceptance criterion cites, rather than several chapters later.
10. Reading locators — every reading annotation names where in the source to look: a section, a figure, a table, a page count. "How to read it" applied to an entire paper is not a locator.
11. Notation registry — every symbol registered in docs/NOTATION.md appears in the chapters it claims to, and vice versa.

make numbers is separate, and closes the gap docs/NUMBERS.md admits to: it recomputes the figures that follow from the reference hardware and the lifecycle model — ridge point, cache sizes, decode ceilings, MFU, lifetime costs, break-even — and checks the prose against them. It caught a decode ceiling in Appendix D that was a factor of two out and contradicted Chapter 13.

The ratchet

Checks 7–10 were added against a book that already fails them, and the fixes are scheduled work (docs/SECOND-EDITION-PLAN.md). So the audit runs as a ratchet: known failures live in tools/audit-baseline.txt and do not break the build, anything new does, and a baselined failure that has been fixed must be struck from the file. The count only goes down.

---

Editing

Edit src/*.html directly; it is the source of truth for both editions. Each chapter is self-contained and follows one shape:

opener (eyebrow · title · one-line blurb)
lead paragraph
  §  prose, with:
       .box.deriv    numbered derivations
       .box.worked   worked examples with real arithmetic
       .box.trap     the ways this material is misread
       .drill        inline drills, answers in <details>
  "What the chapter bought"  — closing synthesis, hands off to the next chapter
  Exercises A (drills) / B (problems) / C (challenge)
  Gate  — what to reproduce on blank paper before continuing
  Readings — each link annotated with how to read it

Write math with \( … \) inline and \[ … \] display. Both editions pick it up automatically.

After editing, run make audit && make pdf.

---

Licence

Split licence — see LICENSE for full text:

• Book content (src/, figures/): CC BY-NC-SA 4.0
• Build tooling (build/, tools/, Makefile, CI): MIT