Skip to content

LiteGPT V2 v2.0.0

Latest

Choose a tag to compare

@kmkrworks kmkrworks released this 06 Nov 02:32
· 11 commits to main since this release

LiteGPT V2: a 201M-parameter language model built from scratch, with its own architecture implementation, tokenizer, training pipeline and weights.

Models

Both load in Hugging Face Transformers with no remote code, and through the offline litegpt Python API and command line in this repository.

Highlights

  • RoPE, RMSNorm with QK-norm, SwiGLU and grouped-query attention, 28 layers × 768
  • A 32,768-token byte-level tokenizer trained on the project's own data mix
  • Clearly better than V1 on five benchmarks and clearly worse on none; about three times better on code text
  • Designed on an RTX 4090 Laptop GPU, pretrained in 37 hours on one A100
  • 32 charts and every run record in the repository

Licence

Code: MIT. Weights: Apache-2.0, each model with its own notice file.

Results and comparisons: docs/results.md