Home

Distributed LLM Training with PyTorch DDP — Wiki

Welcome to the documentation for a runnable data-parallel training demonstration using PyTorch. The project has two clearly separated modes: a small causal Transformer built from scratchand adapter-based fine-tuning of a pretrained Qwen2.5-0.5B-Instruct model.

Scope: This code implements DistributedDataParallel (DDP),tensor parallelism, or an inference-serving system. DDP replicates the model in every worker and averages gradients. The tiny model is not a frontier-scale LLM. GPU/Qwen runs are optional and have been validated as part of this documentation package.

Start here

• Design: System-Architecture, Low-Level-Design, DDP-Gradient-Synchronization
• Model internals: Tiny-Transformer, Qwen-LoRA
• Operate: Installation, Quick-Start, Configuration, Jupyter-and-Colab
• Understand behavior: Data-Pipeline, Runtime-Sequence, Checkpoints, GPU-and-Distributed-Backends
• Quality: Testing, Security-and-Limitations, Troubleshooting, Future-Enhancements

What is inside?

File	Role
llm_ddp_parallel_training.py	Training CLI: tiny causal Transformer or Qwen LoRA
LLM_DDP_Parallel_Training.ipynb	Notebook that writes and launches the training script
llm_ddp_requirements_.txt	Original dependency manifest
requirements.txt	Compatible copy of dependency manifest, added by packaging
README.md	Original project README
diagrams/*.mmd	Standalone editable Mermaid sources, in the code repository
wiki/*.md	GitHub Wiki source pages, published separately

Basic flow

text corpus → IndexedTextDataset → DistributedSampler (per rank)
→ collator → replicated model → forward loss → backward + gradient AllReduce
→ optimizer update → rank-0 checkpoint / LoRA adapters

Validation performed for this package

Tiny mode executed successfully with one CPU worker and two CPU DDP workers, each for two steps. Each run wrote a checkpoint. Qwen LoRA, CUDA and multi-GPU performance have not been tested.

See Publishing-the-Wiki for the separate GitHub code repository and Wiki publication process.