🎉 Modern CUDA Learn Notes with PyTorch: CUDA Cores, Tensor Cores, fp32/tf32, fp16/bf16, fp8/int8, flash_attn, rope, sgemm, hgemm, sgemv, warp/block reduce, elementwise, softmax, layernorm, rmsnorm.

Root Mean Square Layer Normalization

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.

Simple character level Transformer

Generative models nano version for fun. No STOA here, nano first.