BIT is a production-ready, localized transformer model built from scratch. This refactor introduces a BPE tokenizer, KV-cache optimized inference, disk-streaming for large datasets, and a FastAPI serving layer.
- Tokenizer: BPE (Byte Pair Encoding) with a vocabulary size of 10,000.
- Architecture: Transformer with Multi-Head Attention and SwiGLU.
- Inference: Optimized with KV-caching (10x faster) and Top-k/Top-p sampling.
- Training: Early stopping, cosine learning rate annealing, and Tensorboard logging.
- Serving: FastAPI-based REST API for production deployment.
pip install -r requirements.txtpython train.py master_dataset.txtpython chat.py --model bit_model.ptpython -m bit.serve