Repository navigation
Peanut v2.2.0 - LR-aware damping
Peanut v2.2 - LR-aware damping, from community feedback.
What changed from v2.0.0:
- Both dampings now scale with LR. At LR 0.01, damp = 0.2, full. At LR 0.003, damp = 0.66, less. At LR 0.001, damp = 1.0, no damping.
- V inflation also scales with LR.
- Ablation showed neither change alone beats Adam. Both together are needed at high LR.
MLP, 10 epochs, 3 seeds:
- LR 0.003: Peanut v2.2 0.8807 mean, Adam 0.8800 mean
- LR 0.01: Peanut v2.2 0.8619 mean, Adam 0.8613 mean
Community CIFAR-10 probe, 3 epochs, from Hugging Face feedback:
- Peanut beats Adam by 7.04 points at 0.003, 5.80 points at 0.01
- Peanut vs warmup flips with LR
Thanks to Hugging Face community for the ablation and warmup feedback.