Repository navigation
LiteGPT V2: a 201M-parameter language model built from scratch, with its own architecture implementation, tokenizer, training pipeline and weights.
Models
- LiteGPT-V2-Base: the pretrained model, 4,096-token context
- LiteGPT-Llama-V2-Instruct: the chat version
Both load in Hugging Face Transformers with no remote code, and through the offline litegpt Python API and command line in this repository.
Highlights
- RoPE, RMSNorm with QK-norm, SwiGLU and grouped-query attention, 28 layers × 768
- A 32,768-token byte-level tokenizer trained on the project's own data mix
- Clearly better than V1 on five benchmarks and clearly worse on none; about three times better on code text
- Designed on an RTX 4090 Laptop GPU, pretrained in 37 hours on one A100
- 32 charts and every run record in the repository
Licence
Code: MIT. Weights: Apache-2.0, each model with its own notice file.
Results and comparisons: docs/results.md