A ~110M parameter language model trained from scratch using the DeepSeek-V4 architecture. This repo contains all the code, configs, and tokenizer used to pretrain and fine-tune the model.

The model implements the full DeepSeek-V4 feature set at miniature scale:

• Multi-Head Latent Attention (MLA) — 8 heads, 1 KV head (MQA), head_dim=96 (32 RoPE + 64 NoPE), q_lora_rank=160
• Mixture-of-Experts (MoE) — 4 routed + 1 shared expert, top-2 routing, SwiGLU FFN (dim 640)
• Hyper-Connections — hc_mult=4, Sinkhorn routing (2 iterations)
• Multi-Token Prediction (MTP) — 1 next-token prediction layer

├── modeling_deepseek_v4.py         # DeepSeek-V4 model implementation
├── configuration_deepseek_v4.py    # Model config class
├── requirements.txt
├── configs/
│   ├── main_100m.yaml              # Training hyperparameters (100M model)
│   ├── debug.yaml                  # Quick debug config (50 steps)
│   └── fallback_under_1b.yaml      # Alternative config
├── scripts/
│   ├── train_pretrain.py            # Pretraining (SFTTrainer on FineWeb-Edu)
│   ├── train_sft.py                 # SFT fine-tuning (SFTTrainer on SmolTalk)
│   ├── eval_smoke.py                # Perplexity evaluation & generation
│   ├── chat.py                      # Interactive chat
│   ├── upload_to_hub.py             # Hub upload utility
│   ├── count_params.py              # Parameter counting
│   ├── prepare_data.py              # Data preparation
│   └── inspect_deepseek_v4.py       # Architecture inspection
└── tokenizer/
    ├── tokenizer.json
    └── tokenizer_config.json

pip install -r requirements.txt

python scripts/train_pretrain.py --config configs/main_100m.yaml

python scripts/train_sft.py

python scripts/chat.py

python scripts/eval_smoke.py

• bf16 NaN: The model produces NaN in bf16 at this small scale. Use fp32 for inference and training. This is due to the Hyper-Connections architecture producing values that overflow bf16 range.
• from_pretrained quirk: The custom architecture causes from_pretrained to re-initialize some weights. Use manual load_state_dict instead (see model cards for examples).
• Large vocab / small model: The 129K vocab embedding table consumes 37% of all parameters, limiting capacity for language modeling.

MIT