v2.0.2

BetterGPT v2.0.2

This release added a sample text generation code after SFT finetuning also focused on bug fixes for SFT model loading sample generation.

Added

• Added SFT sample generation pipline

Fixed

• Fixed minor bugs identified after the v2.0.1 release.
• Resolved edge cases affecting model and training workflows.

Looking Ahead

Planned work for upcoming releases includes:

• Grouped Query Attention (GQA)
• KV Cache for faster autoregressive inference
• Additional training optimizations
• Evaluation benchmarks and model cards

Thank you to everyone using BetterGPT and contributing feedback.

v2.0.1 - Bug Fixes & Stability Improvements

BetterGPT v2.0.1

This is a maintenance release focused on bug fixes, code quality improvements, and project-wide linting. There are no new features or breaking changes in this release.

Fixed

• Fixed minor bugs identified after the v2.0.0 release.
• Resolved edge cases affecting model and training workflows.
• Improved configuration validation and error handling.

Improved

• Added comprehensive ruff support across the project.
• Refactored code to address linting issues while preserving existing functionality.
• Improved code consistency, readability, and maintainability.
• Updated documentation and inline comments where appropriate.

Developer Experience

• Standardized code style throughout the codebase.
• Reduced lint warnings and improved static code quality.
• Minor internal refactoring to simplify future development and maintenance.

Breaking Changes

None.

Looking Ahead

Planned work for upcoming releases includes:

• Grouped Query Attention (GQA)
• KV Cache for faster autoregressive inference
• Additional training optimizations
• Evaluation benchmarks and model cards

Thank you to everyone using BetterGPT and contributing feedback.

Fine tune pipeline

BetterGPT v2.0.0

This release focuses on making the project fully compatible with the Hugging Face ecosystem while introducing an end-to-end supervised fine-tuning workflow.

What's New

Hugging Face Integration

• Added PreTrainedModel compatibility.
• Added custom PretrainedConfig implementation.
• Full support for AutoModel and AutoConfig.
• Models can now be saved, loaded, and shared using standard Hugging Face APIs.

Supervised Fine-Tuning

• Added an Alpaca instruction fine-tuning pipeline.
• Supports training on instruction-following datasets.
• Includes training scripts and configuration for reproducible experiments.

Improvements

• Cleaner project structure for future model development.
• Better interoperability with Hugging Face tooling and downstream applications.
• Foundation laid for future instruction-tuned and chat-oriented models.

Breaking Changes

• Internal model structure has been updated to align with the Hugging Face PreTrainedModel interface.
• Custom loading workflows from previous versions may require migration to the new API.

Looking Ahead

Planned work for upcoming releases includes:

• Grouped Query Attention (GQA)
• Advanced inference features (KV Cache)
• Expanded training optimizations
• Additional fine-tuning methods (LoRA/QLoRA)
• Evaluation benchmarks and model cards

Thank you to everyone following the project and providing feedback.

v1.0.1 - Bug Fixes & Stability Improvements

v1.0.1

Fixed

• Fixed checkpoint loading edge cases.
• Improved error handling during training.
• Fixed minor configuration validation issues.
• Tokenizer loading

Improved

• Better logging.
• Documentation corrections.

No breaking changes.

v1.0.0

v1.0.0 – Initial Pretraining Release

Initial stable release featuring a complete pretraining pipeline for a decoder-only Transformer. Future releases will add Hugging Face compatibility and fine-tuning support.

Overview

This is the first public release of the project, focused on pretraining a decoder-only Transformer from scratch. The codebase is intended as a clean, educational, and extensible foundation for experimenting with language model training.

Included

• Complete pretraining pipeline
• Custom decoder-only Transformer implementation
• Multi-Head Self-Attention using PyTorch SDPA
• RMSNorm
• RoPE positional embeddings
• SwiGLU feed-forward network
• Weight tying between token embeddings and LM head
• BPE tokenizer training
• Dataset preprocessing and sharding utilities
• Training loop with evaluation, checkpointing, learning rate scheduling, mixed precision, and gradient accumulation
• Text generation with temperature and top-k sampling
• Configuration-driven project structure

Notes

• This release focuses exclusively on pretraining.
• Fine-tuning utilities and Hugging Face compatibility have been removed from this release to keep the codebase focused and easier to understand.
• Hugging Face-compatible model classes and fine-tuning support will be introduced in the next release.

Future Plans (v2)

• Hugging Face PreTrainedModel compatibility
• AutoModel / AutoConfig support
• Fine-tuning pipeline
• Improved checkpoint interoperability
• Additional documentation and examples

Thank you for checking out the project. Feedback, bug reports, and contributions are welcome.