Implement the internals of modern AI systems from scratch — Transformers, vLLM, TRL, and beyond.
Read the paper, then write the code. No GPU required.
76 problems. You write the implementation, a local grading service runs the tests, you see what broke. That's it.
The problems cover what's actually inside Transformers, vLLM, TRL, diffusion models, and GNNs — attention variants, training tricks, inference kernels, alignment algorithms, graph neural networks. No GPU needed.
- Preparing for ML interviews — practice implementing core components under test, not just reading about them
- Learning by building — if you learn best by writing code rather than watching lectures, this is your gym
- Deepening your understanding — you've used
nn.MultiheadAttention, now write it yourself
- Browser editor — Monaco with Python syntax highlighting, no IDE setup
- Instant feedback — submit and see pass/fail per test case in seconds
- Reference solutions — compare after your own attempt
- Progress tracking — solved count and attempt history, persisted across sessions
- AI Help — optional AI-powered hints via any OpenAI-compatible API (configure in
.envor per-user in the UI) - Fully local — nothing leaves your machine (unless you opt into AI Help)
| Layer | Technology |
|---|---|
| Frontend | Next.js + Monaco Editor + Tailwind CSS |
| Backend | FastAPI grading service |
| Judge Engine | torch_judge — executes and validates submissions |
| Storage | SQLite (progress tracking) |
- [2026/04/20] New GNN learning path — 8 problems covering GCN, GAT, GIN, MPNN, GraphSAGE, link prediction, and graph autoencoders. 🔥
- [2026/04/20] New UI redesign with OKLch color system, dark mode, and Geist typography — classic design still available via toggle. 🔥
- [2026/04/13] Submission history — review all your past attempts per problem.
- [2026/04/10] AI Help — optional AI-powered hints via any OpenAI-compatible API. 🔥
- [2026/04/10] Print output capture —
print()statements now show in test results. - [2026/04/09] 68 problems covering Transformers, vLLM, TRL, diffusion models, and more. 🔥
- [2026/04/09] Initial release of Pyre Code 🎉
- Python 3.11+
- Node.js 18+
Option A — one-liner (recommended)
macOS / Linux:
git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
./setup.sh
npm run devWindows (PowerShell):
git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
.\setup.ps1
npm run devWindows (CMD):
git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
setup.bat
npm run devThe setup script automatically creates a .venv Python environment (prefers uv, falls back to python -m venv), installs all dependencies, then prints the start command.
When .venv exists, npm run dev prefers that project-local Python automatically. If .venv is missing, it falls back to the current shell's python.
Option B — conda
git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
conda create -n pyre python=3.11 -y && conda activate pyre
pip install -e ".[dev]"
npm install
npm run dev # run with conda env activatedOption C — manual (venv)
git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
# create a Python env — pick one:
uv venv --python 3.11 .venv && source .venv/bin/activate && uv pip install -e ".[dev]"
# or: python3 -m venv .venv && source .venv/bin/activate && pip install -e ".[dev]"
# Windows: python -m venv .venv && .venv\Scripts\activate && pip install -e ".[dev]"
npm install
npm run devEither way, once running:
- Grading service →
http://localhost:8000 - Web app →
http://localhost:3000
Option D — Docker
git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
docker compose up --buildOpen http://localhost:3000. Progress is persisted in a Docker volume. Run docker compose down -v to reset.
To enable server-side AI hints, copy web/.env.example to web/.env and fill in:
AI_HELP_BASE_URL=https://api.openai.com/v1
AI_HELP_API_KEY=sk-...
AI_HELP_MODEL=gpt-4o-miniAny OpenAI-compatible endpoint works (OpenAI, Anthropic via proxy, Ollama, etc.). Users can also configure their own API key in the UI if no server-side config is set.
76 problems organized by category:
| Category | Problems |
|---|---|
| Fundamentals | ReLU, Softmax, GELU, SwiGLU, Dropout, Embedding, Linear, Kaiming Init, Linear Regression |
| Normalization | LayerNorm, BatchNorm, RMSNorm |
| Attention | Scaled Dot-Product, Multi-Head, Causal, Cross, GQA, Sliding Window, Linear, Flash, Differential, MLA |
| Position Encoding | Sinusoidal PE, RoPE, ALiBi, NTK-aware RoPE |
| Architecture | SwiGLU MLP, GPT-2 Block, ViT Patch, ViT Block, Conv2D, Max Pool, Depthwise Conv, MoE, MoE Load Balance |
| Training | Adam, Cosine LR, Gradient Clipping, Gradient Accumulation, Mixed Precision, Activation Checkpointing |
| Distributed | Tensor Parallel, FSDP, Ring Attention |
| Inference | KV Cache, Top-k Sampling, Beam Search, Speculative Decoding, BPE, INT8 Quantization, Paged Attention |
| Loss & Alignment | Cross Entropy, Label Smoothing, Focal Loss, Contrastive Loss, DPO, GRPO, PPO, Reward Model |
| Diffusion & DiT | Noise Schedule, DDIM Step, Flow Matching, adaLN-Zero |
| Adaptation | LoRA, QLoRA |
| Reasoning | MCTS, Multi-Token Prediction |
| SSM | Mamba SSM |
| Graph Neural Networks | GCN, Graph Readout, GAT, GIN, MPNN, GraphSAGE, Link Prediction, Graph Autoencoder |
Pick one based on what you're working toward:
| Path | Problems | Description |
|---|---|---|
| Transformer Internals | 12 | Activations → Normalization → Attention → GPT-2 Block |
| Attention & Position Encoding | 13 | Every attention variant + RoPE, ALiBi, NTK-RoPE |
| Train a GPT from Scratch | 15 | Embeddings → architecture → loss → optimizer → training tricks |
| Inference & Distributed Training | 9 | KV cache, quantization, sampling, tensor parallel, FSDP |
| Alignment & Agent Reasoning | 6 | Reward model → DPO → GRPO → PPO → MCTS |
| Vision Transformer Pipeline | 7 | Conv → patch embedding → ViT block |
| Diffusion Models & DiT | 5 | Noise schedule → DDIM → flow matching → adaLN-Zero |
| LLM Frontier Architectures | 7 | GQA, Differential Attention, MLA, MoE, Multi-Token Prediction |
| Graph Neural Networks | 8 | GCN → GAT → GIN → MPNN → GraphSAGE → Link Prediction → GAE |
Not sure where to start?
Fundamentals ──→ Transformer Internals ──→ Train a GPT from Scratch
│ │
▼ ▼
Attention & Position Inference & Distributed
│ │
▼ ▼
LLM Frontier Archs Alignment & Reasoning
│
┌───────┼───────┐
▼ ▼ ▼
Vision Trans. Diffusion Graph Neural Networks
| Variable | Default | Description |
|---|---|---|
GRADING_SERVICE_URL |
http://localhost:8000 |
Grading service URL |
DB_PATH |
./data/pyre.db |
SQLite database for progress tracking |
Set in web/.env.local to override.
pyre/
├── web/ # Next.js frontend
│ ├── src/app/ # Pages and API routes
│ ├── src/components/ # UI components
│ └── src/lib/ # Utilities, problem data
├── grading_service/ # FastAPI backend
├── torch_judge/ # Judge engine (problem definitions + test runner)
└── package.json # Dev scripts (runs frontend + backend concurrently)
Contributions are welcome! Here are some ways you can help:
- Submit a new problem — open a PR with the problem definition and test cases in
torch_judge/ - Report a bug — open an issue with steps to reproduce
- Fix a bug — fork, fix, and submit a PR
- Improve docs — typos, clarifications, translations
Please open an issue first for larger changes so we can discuss the approach.
Problem set and judge engine based on TorchCode by duoan, licensed under MIT.
Distributed under the MIT License. See LICENSE for more information.