Architecture Decisions

Architecture Decisions and Trade-offs

ADR-001 — Full-replica DDP rather than sharded training

Decision: use PyTorch DDP with one complete model replica per rank. This makes gradient synchronization and batch partitioning easier to understand and debug. Trade-off: each device must fit the model; this is not an FSDP demonstration.

ADR-002 — Tiny Transformer mode with no network downloads

Decision: provide a 70,978-parameter byte-level causal Transformer for CPU-only demonstrations. This enables two-process verification without CUDA or Hugging Face downloads. Trade-off: the toy corpus and architecture are not a real LLM benchmark.

ADR-003 — Optional pretrained Qwen LoRA mode

Decision: use PEFT LoRA on q_proj and v_proj, retaining pretrained Qwen base weights. Trade-off: adds dependency, device-memory and model-download requirements; not pretraining from scratch.

ADR-004 — Single-file training application

Decision: keep CLI, dataset, collators, modeling, training and saving in one Python module for learning purposes. Trade-off: separate modules, unit tests, data validation and experiment tracking would improve maintainability for larger projects.

ADR-005 — Rank-0 artifact writes

Decision: save outputs on rank 0 and synchronize afterwards. Trade-off: simple write ownership, but no periodic checkpoint/restart support.