C2-Custom-and-Distributed-Training-with-TensorFlow

Course 2: Custom and Distributed Training with TensorFlow

Learn how optimization works and how to use GradientTape and Autograph. Optimize training in different environments with multiple processors and chip types.

Week 1: Differentiation and Gradients

Get an in-depth look at the fundamental building blocks of TensorFlow - tensor objects. Describe the difference between eager mode and graph mode, and explain why eager mode is very user-friendly for developers. Use TensorFlow GradientTape to compute gradients.

• Use various TensorFlow functions to create tensor objects
• Describe the difference between graph-based execution and eager execution in TensorFlow
• Use TensorFlow’s GradientTape to compute derivatives of loss functions

Lab: Basic Tensors
Lab: Gradient Tape Basics
Assignment: Basic Tensor operations and GradientTape

Week 2: Custom Training

Build custom training loops using GradientTape and TensorFlow Datasets for more flexibility in model training.

• Define the steps in a custom training loop
• Implement custom training loops using GradientTape
• Implement a custom training loop with data from TensorFlow Datasets

Lab: Custom Training Basics
Lab: Fashion MNIST using Custom Training Loop
Assignment: Breast Cancer Prediction

Week 3: Graph Mode

Learn about the benefits of generating code that runs in "graph mode". See what graph code looks like, and practice automatically generating this more efficient code using TensorFlow's tools.

• Describe when graph mode code is preferred over eager mode code
• Use decorators and tf.autograph to convert code into graph based code

Lab: Autograph Basics
Lab: Graphs for complex code
Assignment: Horse or Human? In-graph training loop Assignment

Week 4: Distributed Training

Harness the power of distributed training to handle more data and train larger models faster. Get an overview of different distributed training strategies and practice working with two strategies, one that trains on multiple GPU cores and the other that trains on multiple TPU cores.

• Explain how distributed training is different from regular model training
• Use the Mirrored Strategy to train a model on multiple GPUs on the same device
• Use the TPU Strategy to train on multiple cores of a TPU

Lab: Mirrored Strategy Basic
Lab: Multi-GPU Mirrored Strategy
Lab: TPU Strategy
Lab: One Device Strategy
Assignment: Custom training with tf.distribute.Strategy

Reference

Coursera - Custom and Distributed Training with TensorFlow