Skip to content

Peanut v2.2.0 - LR-aware damping

Choose a tag to compare

@dskc dskc released this 14 Nov 15:17
· 5 commits to main since this release

Peanut v2.2 - LR-aware damping, from community feedback.

What changed from v2.0.0:

  • Both dampings now scale with LR. At LR 0.01, damp = 0.2, full. At LR 0.003, damp = 0.66, less. At LR 0.001, damp = 1.0, no damping.
  • V inflation also scales with LR.
  • Ablation showed neither change alone beats Adam. Both together are needed at high LR.

MLP, 10 epochs, 3 seeds:

  • LR 0.003: Peanut v2.2 0.8807 mean, Adam 0.8800 mean
  • LR 0.01: Peanut v2.2 0.8619 mean, Adam 0.8613 mean

Community CIFAR-10 probe, 3 epochs, from Hugging Face feedback:

  • Peanut beats Adam by 7.04 points at 0.003, 5.80 points at 0.01
  • Peanut vs warmup flips with LR

Thanks to Hugging Face community for the ablation and warmup feedback.