Repository navigation
Emerging Optimizers v0.5.0
Highlights
- New
emerging_optimizers.shampoopackage: Shampoo (#287), KL-Shampoo (#302), and a rewritten SOAP family (KlSoapV3,ReklsV3,KlMSoap) built on a shared preconditioner base (#281). Shampoo's update scaling has been improved (#306), KL-Shampoo now starts its eigenvalues at 1 (#310), and inverse matrix roots can now be computed with CANS, including a dedicated 4th-root inverse (#321). - Muon and SOAP defaults now follow the SOAP, Muon, and Beyond recipe (arXiv:2607.20548) (#319). This changes training behavior; see Breaking changes.
- Pluggable weight-update hooks: the new
emerging_optimizers.weight_update_hooksmodule providesHyperballHook,RadialBrakeHookandRelativeUpdateHook, passed to orthogonalized optimizers withweight_update_hook=(#224). - More Riemannian optimizers: oblique steepest-descent (LMO) optimizers
oblique_steepest_sgdandoblique_steepest_adam, plus RMS scaling (#299); Polar Express Newton–Schulz as a retraction forIso(#294); scaled-Stiefel support inSpel(#301). - New examples and docs: a minimal-dependency nanoGPT training example (#308), a primer on Newton–Schulz orthogonalization quality (#309), a README section on resuming bf16 runs from optimizer checkpoints (#324), and an API reference that now covers Shampoo, SOAP v3, the weight-update hooks, Spel, the Stiefel retractions, StackedSoap, MOSO, TpRekls and PSGDPro (#327).
Breaking changes
-
The
soappackage is nowlegacy_soap(#279). Update imports fromemerging_optimizers.soaptoemerging_optimizers.legacy_soap. The registry names (soap,rekls,stacked_soap,tp_rekls,moso) are unchanged. -
New Muon defaults (#319):
Argument v0.4.0 v0.5.0 momentum0.95 0.9 weight_decay0.01 0.1 coefficient_type"quintic""polar_express"num_ns_steps5 8 extra_scale_factor1.0 0.2 -
New legacy SOAP-family defaults (#319):
weight_decaygoes from 0.01 to 0.1 inSOAP,REKLS,StackedSoapandTpRekls, andSOAPnow defaults touse_kl_shampoo=True.
To keep the v0.4.0 behavior, pass the old values explicitly. -
Weight-update hooks replace subclass hooks (#224).
OrthogonalizedOptimizerno longer calls thepre_weight_update_fn_inplace/post_weight_update_fn_inplacemethods. Subclasses that overrode them should implement aWeightUpdateHookand pass it asweight_update_hook=.
New features
- Sinkhorn balancing utilities are back as
orthogonalized_optimizers.sinkhorn_utils.sinkhorn_balance; they were removed in v0.4.0 (#298). - Batched SYRK Triton kernel (#276, #286).
- Muon now exposes its Newton–Schulz settings, such as
num_ns_steps, as properties (#283).
Bug fixes
- Polar Express coefficients are now correct for runs with more than 8 steps (#303).
- The all-reduce in tensor-parallel Newton–Schulz now runs at the right precision (#304).
- Tensor-parallel paths fall back to the single-GPU code when
tp_grouphas size 1 (#314). AdaptiveMuon.stepnow handles parameter groups added after the optimizer was created (#313).- Corrected the Riemannian gradient in
Spel(#301) and the bias correction inObliqueAdam(#299). - Registry argument validation for
muon_hyperballis fixed (#312). PSGDPronow validates its hyperparameters when it is constructed (#317).- The
Args:sections of the orthogonalized optimizers' docstrings render correctly on Python 3.13 (#327).