Skip to content

v0.5.0

Latest

Choose a tag to compare

@nemo-automation-bot nemo-automation-bot released this 10 Nov 19:41
78d0548

Emerging Optimizers v0.5.0

Highlights

  • New emerging_optimizers.shampoo package: Shampoo (#287), KL-Shampoo (#302), and a rewritten SOAP family (KlSoapV3, ReklsV3, KlMSoap) built on a shared preconditioner base (#281). Shampoo's update scaling has been improved (#306), KL-Shampoo now starts its eigenvalues at 1 (#310), and inverse matrix roots can now be computed with CANS, including a dedicated 4th-root inverse (#321).
  • Muon and SOAP defaults now follow the SOAP, Muon, and Beyond recipe (arXiv:2607.20548) (#319). This changes training behavior; see Breaking changes.
  • Pluggable weight-update hooks: the new emerging_optimizers.weight_update_hooks module provides HyperballHook, RadialBrakeHook and RelativeUpdateHook, passed to orthogonalized optimizers with weight_update_hook= (#224).
  • More Riemannian optimizers: oblique steepest-descent (LMO) optimizers oblique_steepest_sgd and oblique_steepest_adam, plus RMS scaling (#299); Polar Express Newton–Schulz as a retraction for Iso (#294); scaled-Stiefel support in Spel (#301).
  • New examples and docs: a minimal-dependency nanoGPT training example (#308), a primer on Newton–Schulz orthogonalization quality (#309), a README section on resuming bf16 runs from optimizer checkpoints (#324), and an API reference that now covers Shampoo, SOAP v3, the weight-update hooks, Spel, the Stiefel retractions, StackedSoap, MOSO, TpRekls and PSGDPro (#327).

Breaking changes

  • The soap package is now legacy_soap (#279). Update imports from emerging_optimizers.soap to emerging_optimizers.legacy_soap. The registry names (soap, rekls, stacked_soap, tp_rekls, moso) are unchanged.

  • New Muon defaults (#319):

    Argument v0.4.0 v0.5.0
    momentum 0.95 0.9
    weight_decay 0.01 0.1
    coefficient_type "quintic" "polar_express"
    num_ns_steps 5 8
    extra_scale_factor 1.0 0.2
  • New legacy SOAP-family defaults (#319): weight_decay goes from 0.01 to 0.1 in SOAP, REKLS, StackedSoap and TpRekls, and SOAP now defaults to use_kl_shampoo=True.
    To keep the v0.4.0 behavior, pass the old values explicitly.

  • Weight-update hooks replace subclass hooks (#224). OrthogonalizedOptimizer no longer calls the pre_weight_update_fn_inplace / post_weight_update_fn_inplace methods. Subclasses that overrode them should implement a WeightUpdateHook and pass it as weight_update_hook=.

New features

  • Sinkhorn balancing utilities are back as orthogonalized_optimizers.sinkhorn_utils.sinkhorn_balance; they were removed in v0.4.0 (#298).
  • Batched SYRK Triton kernel (#276, #286).
  • Muon now exposes its Newton–Schulz settings, such as num_ns_steps, as properties (#283).

Bug fixes

  • Polar Express coefficients are now correct for runs with more than 8 steps (#303).
  • The all-reduce in tensor-parallel Newton–Schulz now runs at the right precision (#304).
  • Tensor-parallel paths fall back to the single-GPU code when tp_group has size 1 (#314).
  • AdaptiveMuon.step now handles parameter groups added after the optimizer was created (#313).
  • Corrected the Riemannian gradient in Spel (#301) and the bias correction in ObliqueAdam (#299).
  • Registry argument validation for muon_hyperball is fixed (#312).
  • PSGDPro now validates its hyperparameters when it is constructed (#317).
  • The Args: sections of the orthogonalized optimizers' docstrings render correctly on Python 3.13 (#327).