DDP Gradient Synchronization

DistributedDataParallel — Gradient Synchronization

What DDP actually does

The same model parameters are replicated on each rank. Each rank sees different examples chosen by DistributedSampler and computes a local loss. During backward, DDP's gradient synchronization (typically bucketed AllReduce) ensures ranks receive consistent averaged gradients before their optimizer updates.

For R ranks with per-rank gradients g_r (under DDP's usual reduction behavior), the synced gradient is approximately:

g_synced = (1/R) * sum(g_r for r in ranks)

The implementation performs an additional dist.all_reduce on a detached loss scalar after the optimizer update, solely to display mean training loss. Do not confuse that reporting collective with gradient synchronization.

Effective batch size

For this project, with no gradient accumulation:

effective_global_batch = --batch-size × WORLD_SIZE

For example, two ranks with --batch-size 2 process four training examples per synchronous optimizer step, subject to sampler padding and end-of-epoch behavior.

Important constraints

• Model parameters, optimizer state and activations must fit on each GPU; this is not FSDP.
• DistributedSampler may duplicate indices when the dataset length is not evenly divisible by world size.
• Each rank must execute the same number of training steps and collectives.
• sampler.set_epoch(epoch) changes the shuffle seed each epoch.
• The code does not implement gradient accumulation, ZeRO, or fault-tolerant recovery.

See Data-Pipeline and GPU-and-Distributed-Backends.