Quick Start

Quick Start — Offline DDP and Optional GPU LoRA

Run commands from the code-repository root, where llm_ddp_parallel_training.py exists.

1. One CPU process, no model download

python llm_ddp_parallel_training.py --model tiny --cpu --max-steps 2 --save-dir ddp_example_output

2. Two CPU processes, real distributed gradient synchronization

python -m torch.distributed.run --standalone --nproc_per_node=2 llm_ddp_parallel_training.py --model tiny --cpu --batch-size 2 --max-steps 2

Expected pattern (exact loss values can vary):

Mode=tiny, world_size=2, backend=gloo, per_device_batch=2, global_batch=4
Rank 0/2 on cpu: first batch has sample IDs [...]
Rank 1/2 on cpu: first batch has sample IDs [...]
epoch=1 step=1 mean_loss=...
Tiny Transformer saved at: .../tiny_transformer.pt

3. Optional Qwen LoRA on one GPU

python -m torch.distributed.run --standalone --nproc_per_node=1 llm_ddp_parallel_training.py --model qwen --batch-size 1 --seq-len 64 --max-steps 2

This command downloads pretrained weights and requires compatible CUDA hardware and dependencies. To use two CUDA GPUs, confirm torch.cuda.device_count() >= 2 and change --nproc_per_node=2. Do not run more processes than devices.

4. Custom corpus

python llm_ddp_parallel_training.py --model tiny --cpu --corpus-file examples/sample_corpus.txt --repeat 1 --max-steps 2

See Configuration and Troubleshooting.