smol-glm-5.2 TinyStories checkpoint

smol-glm-5.2 — TinyStories checkpoint

The first trained release of smol-glm-5.2, a readable 50.15M-parameter
miniature of the GLM-5.2 language-model architecture with approximately
21.77M parameters active per token.

Training result

• 10,000 optimizer steps on TinyStories
• selected checkpoint: step 9,750
• best validation loss: 1.6653
• final validation loss: 1.6774
• final next-token train loss: 1.5918
• final measured throughput: 8,199 tokens/s
• peak allocated VRAM: 3.86 GiB
• trained locally on an RTX 4060 Laptop GPU with 8 GB VRAM

Architecture

• 13 decoder layers with low-rank Multi-head Latent Attention
• DeepSeek Sparse Attention with causal Top-64 lightning indexing
• GLM-5.2 FFFSSSFSSSFSS IndexShare schedule
• 16 routed experts, Top-2 per token, plus one shared expert
• parameter-shared multi-token prediction across three future depths
• Muon Split for matrices and AdamW for embeddings, norms, and scalars

Bundle

• model.safetensors — 50,153,344-parameter BF16 inference weights without optimizer state
• config.json — exact model architecture
• tokenizer.json — 8,192-token TinyStories BPE tokenizer
• training_metadata.json — selected checkpoint, training configuration, and final run state
• sha256sums.txt — integrity checksums for every artifact

Download and generate

mkdir -p pretrained/smol-glm-5.2

gh release download v0.1.0-tinystories \
  --repo cneuralnetwork/smol-glm-5.2 \
  --dir pretrained/smol-glm-5.2

python3 generate.py pretrained/smol-glm-5.2/model.safetensors \
  --prompt "Once upon a time, a little fox"

Example from the exported model:

Once upon a time, a little fox was walking in the woods when he stumbled on
a piece of cardboard. He was very sad because he could not find any friends.
He looked around, but the cardboard wasn't there.