AI Systems Overview

• Measure goodput: More than FLOPS or utilization, measure the ratio of time the GPU spends performing useful work (e.g., forward/backprop computations) vs waiting on data or other overhead as goodput focuses on effective, useful GPU utilization. To measure this ratio use
  • Use NVIDIA Nsight Systems/Compute
  • PyTorch profiler
• Prefer skillful engineering optimizations instead of brute-force spending: Clever software and system optimizations can bridge the gap when hardware is limited, enabling results that would otherwise require far more expensive infrastructure (DeepSeek’s achievement).
• Look for order-of-magnitude impact with incremental optimizations: At scale, small-percentage efficiency gain can save millions of dollars. Avoid redundant computations and slow data pipelines.
• Approach performance tuning with a profile-driven mindset: Use profilers to identify and mitigate the true bottlenecks (compute utilization, memory bandwidth, memory latency, cache misses, or communication/network delays).
• Maintain a holistic view: The best performance engineers consider hardware-software codesign: sometimes algorithm changes can alleviate hardware limits, and sometimes new hardware features enable new algorithms.
• Stay informed on the latest hardware, software, and algorithms: Modern AI hardware and software are evolving rapidly use, stay on top ([MLPerf ])https://mlcommons.org/benchmarks/training/))