Roadmap

Possible Future Enhancements (Not Implemented)

These are proposals, not current features:

1. Add a standalone inference CLI that loads checkpoint metadata, tokenizer and frozen embedding matrix.
2. Add unit tests for attention masking, shape validation, SDPA parity, and checkpoint restoration.
3. Track full training configuration and package versions to improve reproducibility.
4. Introduce deterministic seeding options and structured experiment logging.
5. Add genuine validation curves and measured CPU/CUDA benchmarks, including max memory.
6. Add evaluation on the official complete Yelp test split and class-wise precision/recall/F1.
7. Add configurable data sampler and a robust empty-batch / all-padding policy.
8. Explore a separate generative-LM project (causal decoder, next-token objective, autoregressive inference) if the goal is really LLM-from-scratch.
9. Consider GPU mixed precision, gradient accumulation, DDP or FSDP only after testing and verifying each path.
10. A new small GPT-style decoder-only model will be built from scratch in the next iteration.