Skip to content

v1.0.0

Choose a tag to compare

View all tags
@harikrish2727 harikrish2727 released this 01 Jul 20:00
· 40 commits to main since this release
v1.0.0
a92e54e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v1.0.0 – Initial Pretraining Release

Initial stable release featuring a complete pretraining pipeline for a decoder-only Transformer. Future releases will add Hugging Face compatibility and fine-tuning support.

Overview

This is the first public release of the project, focused on pretraining a decoder-only Transformer from scratch. The codebase is intended as a clean, educational, and extensible foundation for experimenting with language model training.

Included

  • Complete pretraining pipeline
  • Custom decoder-only Transformer implementation
  • Multi-Head Self-Attention using PyTorch SDPA
  • RMSNorm
  • RoPE positional embeddings
  • SwiGLU feed-forward network
  • Weight tying between token embeddings and LM head
  • BPE tokenizer training
  • Dataset preprocessing and sharding utilities
  • Training loop with evaluation, checkpointing, learning rate scheduling, mixed precision, and gradient accumulation
  • Text generation with temperature and top-k sampling
  • Configuration-driven project structure

Notes

  • This release focuses exclusively on pretraining.
  • Fine-tuning utilities and Hugging Face compatibility have been removed from this release to keep the codebase focused and easier to understand.
  • Hugging Face-compatible model classes and fine-tuning support will be introduced in the next release.

Future Plans (v2)

  • Hugging Face PreTrainedModel compatibility
  • AutoModel / AutoConfig support
  • Fine-tuning pipeline
  • Improved checkpoint interoperability
  • Additional documentation and examples

Thank you for checking out the project. Feedback, bug reports, and contributions are welcome.

Assets 2
Loading