Future Enhancements

Future Enhancements — Not Implemented

The following are potential upgrades, not existing features:

1. FSDP2 / sharded parameters and optimizer states for lower per-GPU model-memory pressure.
2. Gradient accumulation to change effective global batch size without increasing per-device microbatches.
3. True mixed precision with autocast / bf16 and tested optimizer/gradient-scaling behavior.
4. Validation and evaluation harness with held-out perplexity and useful downstream metrics.
5. Instruction fine-tuning with structured prompt/answer pairs and response-only label masking.
6. Resumable checkpoints that save optimizer, scheduler, RNG, step and rank metadata.
7. Profiling and tracking of per-rank throughput, communication, VRAM and GPU utilization.
8. Distributed fault tolerance and robust data streaming for larger datasets.
9. Automated CI for the CPU two-rank smoke test and linting.

Do not label these as supported until implementation and testing are complete.