├── README.md # 项目总览和章节索引
├── Chapter1/ # Chapter 1：Gradient Descent & Backpropagation
│ ├── img/ # 章节相关图片资源
│ ├── README.md # 章节详细内容
│ └── references.md # 章节参考资料链接
├── Chapter2/ # 其他章节遵循相同结构
│ ├── img/
│ ├── README.md
│ └── references.md
├── ...
- 章节目录：使用
Chapter{N}/格式，其中{N}为章节编号（如Chapter1/,Chapter2/） - 资源目录：使用小写字母和下划线（
img/） - 文件命名：
- 章节详细内容文件统一使用
README.md - 章节参考资料文件统一使用
references.md - 图片文件使用描述性名称，支持中文和英文（如
BP_in_BN.png）
- 章节详细内容文件统一使用
-
章节README结构：
# [章节标题] ## 概述 ## 理论基础 ## 实践应用 ## 参考资料
-
图片资源：
- 每个章节的图片存放在对应章节的
img/目录下 - 图片引用使用相对路径：
- 每个章节的图片存放在对应章节的
-
添加新章节：
- 在根目录下创建
Chapter{N}/目录 - 创建
Chapter{N}/README.md,Chapter{N}/references.md和Chapter{N}/img/目录 - 在主
README.md中添加章节详细内容大纲
- 在根目录下创建
-
内容更新：
- 章节详细内容更新在对应章节的
README.md中 - 主
README.md保持章节概览和索引功能 - 保持目录结构扁平化，避免过深嵌套
- 章节详细内容更新在对应章节的
- 使用 Git 进行版本控制
- 提交信息格式：
<type>: Chapter{N} <description> - 大文件（PDF、大图片）考虑使用 Git LFS
- Gradient Descent & Backpropagation
- Deep Neural Networks & Regularization
- Transformers
- LLMs
- Multimodal Transformers
- Reinforcement Learning
- Mixture of Experts (MoE)
- Diffusion Models
- AI Infrastructure (GPUs & Parallelism)
- Emerging AI architecture (Mamba, RetNet, RWKV, etc)
- Reverse-mode autodiff vs. forward-mode; computational graphs; the chain rule in practice.
- From scalar backprop to vectorized backprop; Jacobian–vector products; gradient checking.
- Vanishing/exploding gradients, initialization & stabilization (layer norm preview).
- Derive backprop for a linear layer + nonlinearity; show how grad_fn chains compose.
- Dissect PyTorch’s autograd tape on toy MLP/CNN; write a custom autograd.Function (e.g., stable log-sum-exp).
- Implement a tiny autodiff engine to demystify backward(); extend to vector operations.
- Profiling and debugging gradients (anomaly mode, hooks, in-place gotchas).
- Mini-project: “Rebuild” a 2-layer net in ~100 lines that matches PyTorch’s gradients on CIFAR-10 subset.
- PyTorch Autograd mechanics & blog
- CS231n backprop notes
- Karpathy’s micrograd
- BatchNorm; Mixup/CutMix.
- Bias–variance, L2/weight decay as constrained optimization; early stopping.
- Data augmentation.
- Dropout (intuition + model averaging view); label smoothing.
- Batch norm mechanics & pitfalls.
- Mixup/CutMix as vicinal risk minimization; when they help/hurt.
- Ablation lab: train a compact ResNet/MLP and toggle regularizers.
- Implement batch-norm from scratch; compare to PyTorch; study train/test stats drift.
- Augmentations: reproduce mixup/CutMix gains on CIFAR-10/100 with a fixed budget.
- Goodfellow Ch.7
- Scaled dot-product attention; encoder vs. decoder; pre-LN vs. post-LN.
- Positional encodings (absolute, learned, RoPE).
- Depth/width/heads trade-offs; residual pathways & normalization for stability.
- Pre-LN/LayerNorm analysis; FlashAttention.
- Implement a minimal decoder block.
- Add RoPE; benchmark FlashAttention kernels vs. baseline.
- Long-context tricks: key-value caching, sliding-window attention.
- Reproduce a small Transformer on WikiText-2; report perplexity vs. context length.
- PyTorch blog on FlashAttention-3
- Why decoder-only for generative LLMs (causal LM & in-context learning); GPT-3 scaling.
- Alignment overview: supervised fine-tuning (SFT) vs. RLHF; InstructGPT methodology.
- Parameter-efficient tuning: LoRA core idea + variants (LoRA+).
- Build an SFT pipeline (Transformers Trainer) on a domain dataset.
- Fine-tune with LoRA (HF PEFT); measure memory/throughput vs. full-fine-tune.
- Prompting vs. SFT vs. LoRA ablation on the same tasks (e.g., summarization, QA).
- Build a LLM from scratch (Raschka)
- HuggingFace Transformers docs
- PEFT LoRA guide
- Vision Transformer (ViT): patchification, class token, pretrain-then-fine-tune; Swin’s shifted windows.
- Self-supervised ViTs: DINO; masked autoencoders (MAE).
- CLIP: contrastive image–text pretraining; zero-shot transfer.
- Train a tiny ViT or Swin on a small dataset; compare to CNN baseline.
- Zero-shot classification with CLIP; prompt/adapter tuning; retrieval demo.
- Mini-survey: “When does MAE beat supervised pretraining?” (reproduce a small ablation).
- Swin
- DINO
- MAE
- CLIP
- On-policy policy-gradient recap.
- TRPO vs. PPO clips/KL penalties.
- PPO for language models (as in RLHF); monitoring KL, clip fraction, reward scaling.
- GRPO (Group Relative Policy Optimization): critic-free baselining from grouped scores; pros/cons.
- Classic RL lab: PPO on CartPole/LunarLander (Gymnasium or SB3); visualize learning dynamics.
- LLM-RL lab: TRL PPO or GRPO on a tiny text task (rule-based reward, e.g., length/regex or BLEU).
- Stress-test hyper-params (entropy bonus, KL-coef); “RL-for-LLMs” pitfalls.
- EasyRL
- PPO (Schulman et al.)
- GRPO (DeepSeekMath)
- Gymnasium
- Conditional computation; top-k routing; load-balancing losses; expert capacity & token dropping.
- From Shazeer MoE to GShard to Switch Transformer; stability concerns & overflow.
- Compute vs. params: why sparse activation scales cheaply.
- Train a toy MoE layer; visualize gate distribution & token-to-expert traffic.
- Expert parallelism with DeepSpeed-MoE; compare throughput to dense baselines.
- Research sketch: dynamic routing with curriculum or domain-specialized experts.
- DeepSpeed MoE docs/tutorials
- DDPM forward/reverse processes; reweighting & variance schedules; likelihood vs. sample quality.
- DDIM accelerated sampling; classifier-free guidance; score-based SDE perspective.
- Latent diffusion (LDM/Stable Diffusion) and why latent spaces matter.
- Implement a minimal U-Net + DDPM on MNIST/CIFAR-10; add DDIM sampling head.
- Add classifier-free guidance, measure FID vs. sampling steps.
- Latent-diffusion lab: fine-tune a tiny LDM on a narrow concept; log speed/VRAM vs. pixel-space DDPM.
- DDPM
- DDIM
- Classifier-Free Guidance
- Score-SDE
- LDM
- GPU architecture essentials: tensor cores, memory hierarchy, mixed precision, NCCL collectives.
- Parallelism taxonomy: data, tensor (intra-layer), pipeline (GPipe), and ZeRO sharding; FSDP vs. DeepSpeed ZeRO.
- System-level throughput math: utilization, overlap (compute/comm), activation checkpointing.
- Scale a small Transformer with: (a) DDP baseline, (b) FSDP shard, (c) tensor + pipeline (Megatron/DeepSpeed); compare tokens/sec & cost.
- Memory-budgeting lab: fit the largest possible model on a single GPU via mixed precision + activation checkpointing; document trade-offs.
- Write a reproducibility one-pager: seeds, determinism, logging, experiment tracking.
- Programming Massively Parallel Processors (Wen-mei W. Hwu)
- NVIDIA A100 whitepaper
- Megatron-LM scaling case study
- PyTorch FSDP tutorials