v0.9.1

GeometricOptimizers v0.9.1

Diff since v0.9.0

Not a breaking release. Full notes: CHANGELOG.md, section 0.9.1.

Added

• CompositeMethod(; manifold, array): one optimizer method for the manifold leaves and one for the plain-array leaves of a parameter set. A TrainingOptimizer steps it as one optimizer per leaf.
• A TrainingOptimizer reports its retraction to a step observer.
• scripts/retraction_records.jl: the retraction benchmark as machine-readable CSV records.

Fixed

• Files that tag a leaf LowerTriangular or UpperTriangular load again. 0.9.0 registered only the renamed StrictlyLowerTriangular and StrictlyUpperTriangular, so files written before 0.9 failed to load.
• An out-of-range index on a structured matrix raises instead of returning 0.0.
• retraction_differential(::Cayley, B, α) runs on a device without scalar indexing.
• Ω no longer computes an unused matrix.
• The two-argument NewtonOptimizerCache constructor, which no call site reached, is deleted.

Changed

• Fewer allocations in a manifold solver step, in the four dense-to-packed projections, in symplectic_gram_schmidt!, sr! and the apply_S_*! functions, and in *(::SkewSymMatrix, ::SkewSymMatrix) on the host.
• The [compat] floor of NeuralNetworkParameters is 0.5.3.

Merged pull requests:

• Add CompositeMethod: one TrainingOptimizer with a cache and a state per leaf (#121) (@benedict-96)
• Record the retraction benchmark as CSV, from one invocation per row (#122) (@benedict-96)
• Register the pre-0.9 names of the triangular types for loading (#146) (@benedict-96)
• Let a TrainingOptimizer report its retraction to a step observer (#147) (@benedict-96)
• Test the training step's container gradient and step-size conversion (moved from GeometricMachineLearning) (#159) (@michakraus)
• Cut the per-step allocations of a manifold solve, and close the audit findings (#160) (@michakraus)
• Rename a citation key to the house scheme (#162) (@michakraus)

v0.9.0

GeometricOptimizers v0.9.0

Diff since v0.8.0

GeometricOptimizers 0.9 is the optimizer that GeometricMachineLearning trains with. The full list, with the evidence for each entry, is in CHANGELOG.md.

Breaking Changes

• The methods and schedules carry no element type: Adam(), MomentumMethod(; α), DecayingStatic(; η₁, η₂, n) etc.; Adam(Float32) is a MethodError. Optimizer and TrainingOptimizer convert the method to the element type of the parameters.
• One allocator convention: zeros([backend,] X{T}, dims...) and rand([rng,] [backend,] X{T}, dims...); the order rand(backend, rng, X, ...) is removed.
• LowerTriangular/UpperTriangular are renamed StrictlyLowerTriangular/StrictlyUpperTriangular; NewtonOptimizerState is renamed NewtonState.
• gradient, value and check are public, no longer exported. StiefelProjection takes (backend, T, N, n) or (T, N, n).
• vec of a structured matrix or lift is Base's; use freeparameters or parent for the storage. LazyArrays is no longer a dependency.
• A parameter set has no default gradient: pass ∇F!, or mode = :autodiff. No default gradient on a device.
• HessianBFGS, HessianDFP, IterativeHessian, AdamW and isaOptimizerState are deleted.
• zero(Y) of a Stiefel or Grassmann point is a plain zero matrix. A state reused for a second solve! starts again.
• rgrad on a device refuses a gradient on another backend; an operand without a backend raises in checked +, -, *, mul!.
• Requires Julia 1.12 and NeuralNetworkParameters 0.5.

Added

• TrainingOptimizer and optimization_step!: one training step per minibatch with a caller-computed gradient. PrecomputedGradient, default_step_size; linesearch also takes a number.
• solve! runs on a device with GradientMethod, MomentumMethod, Adam, BFGS and DFP; a Metal workflow and Metal tests.

Changed

• An Adam step updates its moments in place: 0 B per step on a dense parameter set (was 0.67 MB in Float32).
• The retraction workspace removes the allocations of the geodesic, the Cayley differential and the BFGS/DFP section update.
• Every numeric test runs in Float32 and Float64.

Fixed

• global_section is orthogonal to the point on every draw; Δf spans one step for every method; the state after solve! holds the returned iterate; every product among the owned matrix types runs on a device.

Merged pull requests:

• Use transpose in the symplectic Householder; pin ProjectTo (#111) (@michakraus)
• Route *, + and - through one Union design (G1) (#112) (@michakraus)
• Run owned products and sums on a device (G2) (#113) (@michakraus)
• Build a wide StiefelProjection on a device (#115) (@michakraus)
• Seed the device sweep; test the backend constructor (#116) (@michakraus)
• Run the device sweep on Metal on Apple silicon and in a macos-15 job (#117) (@michakraus)
• Orthonormalise a global section twice (#118) (@michakraus)
• Archive the global_section probes in scripts/ (#119) (@michakraus)
• One allocator chain, the renames, and public instead of exported (G3) (#120) (@michakraus)
• Move the training step into GeometricOptimizers (G4) (#124) (@michakraus)
• Reuse solve!'s objective value in the state update (#125) (@michakraus)
• Move the Open Issues section of the CHANGELOG into KNOWN_ISSUES.md (#126) (@michakraus)
• Migrate the test suite to the shared layout (part M6) (#132) (@michakraus)
• Optimizer-state semantics: advance the state before the stop check, one quasi-Newton cache (#133) (@michakraus)
• Assert predicates directly in tests (#134) (@michakraus)
• Clear the fatou findings and add the ExplicitImports guard (#135) (@michakraus)
• Clear the two unused-binding lint findings in the tests (#136) (@michakraus)
• Add test/quality/jet.jl for the hot and kernel paths (part J2) (#137) (@michakraus)
• Delete dead and error-only code, and route the elementwise primitives through mapstorage! (part G6) (#138) (@michakraus)
• Raise the KernelAbstractions and ChainRulesCore compat floors (#139) (@michakraus)
• Run the Metal tests in a default run on Apple silicon, with a visible skip where Metal is not functional (#140) (@michakraus)
• Run solve! on a device backend (part G7) (#141) (@michakraus)
• Link the manual's TikZ figures from GeometricFigures (part C2) (#142) (@michakraus)
• Raise the Julia floor to 1.12 (#143) (@michakraus)
• perf: in-place geodesic, Cayley differential and section transport (G8) (#144) (@michakraus)
• Move the top-level test files to the directories they test (#145) (@michakraus)
• Move the four core test files above the 60 s cold budget to slow (#148) (@michakraus)
• CI: coverage from Julia 1 on ubuntu, save the cache only from a job that succeeds (#149) (@michakraus)
• Move the doctests into their own test group (#150) (@michakraus)
• Record the Downgrade job's LLVM and Metal conflict in KNOWN_ISSUES.md (#151) (@michakraus)
• Run every numeric test in Float32 and Float64 (#156) (@michakraus)
• Allocation-free Adam step, no default gradient for a parameter set, NNP 0.5 (G11) (#157) (@michakraus)
• Prose pass: no history in comments and docstrings (G10) (#158) (@michakraus)

Closed issues:

• Do we need both Δf and Δf̃? (#6)
• Implement a DataLoader (and the rest of the neural network scaffolding) properly instead of duplicating it in scripts (#15)
• Format the whole repository with JuliaFormatter and add a format check to CI (#19)
• Dead code: unreachable compute_direction!, duplicated _copyto!, exported-but-undefined optimizer names (#20)
• Base.zero(::StiefelManifold) returns a StiefelLieAlgHorMatrix, i.e. a different type than its argument (#21)
• _fill! on a Manifold is a silent no-op, so GradientCache(x::Manifold) is not poisoned with NaN (#22)
• mode = :finitediff throws a MethodError on NamedTuple parameters (#24)
• error(ps::NamedTuple) in test/optimizer_convergence/svd_optim.jl shadows Base.error (#26)
• Bare Manifold parameters: mode = :finitediff and default_gradient have no Manifold method (#27)
• The native 𝔄 algorithms allocate a fresh temporary per operation, and the ordering is inverted (#77)
• cayley docstring prints the wrong formula (the code is correct) (#81)
• solution and gradient on NewtonOptimizerState return the previous iterate's quantities (#106)
• A state read after solve! lags the returned iterate by one, and the Newton pair collapses (#108)

v0.8.0

GeometricOptimizers v0.8.0

Diff since v0.7.0

This release makes the host/device seam and the LinearAlgebra contract explicit. The full
changelog for 0.8.0 is in CHANGELOG.md.

Breaking Changes

• A computation across two backends is refused by name. Most were not refused before: seven of
  twelve mixed operations returned an answer, and which backend it landed on depended on the
  argument order. Move one operand first, with copyto! or changebackend.
• broadcast(f, Y) on a Manifold returns a plain array. The method that rewrapped the result
  is deleted, because the wrapper claimed an invariant the result does not hold. Write
  manifold_constructor(Y)(broadcast(f, Y.A)) if you need the wrapper back. Dot syntax never
  reached the deleted method and is unaffected.
• An allocator that names a backend refuses an element type that backend cannot hold, instead
  of failing later inside a kernel.
• Optimizer has ten type parameters, against eight in 0.7.0 —
  Optimizer{T, ALG, OBJ, GT, HT, OCT, LST, RT, WT, OT}, adding one for the observer and one,
  WT, for the retraction workspace. Constructors and accessors are unaffected.
• mul! and copyto! return their destination, and A * b returns a vector, for
  SkewSymMatrix, SymmetricMatrix and both triangulars.
• assign! and copyto! reject a mismatched pair instead of writing part of the destination.
• SkewSymMatrix and SymmetricMatrix project an integer matrix into float(T), not into
  Float32.
• Points and global sections are orthonormalized with CholeskyQR2, not LinearAlgebra.qr!.
  This is what makes a draw on a device work; the factor differs in the last digits.
• StiefelProjection's N and n are Int, not Integer.
• A triangular product whose operands differ in storage or element type returns a triangular,
  where it returned a dense Matrix.
• Newton rejects a Manifold solution and a parameter set, with a message naming the method.
• solve! does not evaluate the objective twice at an iterate it has just evaluated.

New Features

• The symplectic Stiefel manifold, and the symplectic SR decomposition behind it.
• Manifold points, global sections and horizontal lifts run on a GPU backend.
• Every optimizer state reports the objective values it holds.

Bug Fixes

• A manifold step no longer allocates per iteration.
• cayley and the symplectic Stiefel metric drop their N^3 term.
• Nine expressions wrote adjoint where the identity is a transpose, which gave a complex
  triangular multiply the wrong answer.
• A row vector times one of this package's matrix types is an ordinary product again.
• Ambiguities between two of this package's own methods are resolved.

Known limitation

rgrad accepts an ambient gradient that is not on its point's backend and moves it across. This
is a temporary shim for defects in GeometricMachineLearning.jl and AbstractNeuralNetworks.jl,
and it is to be reverted once those close.

Merged pull requests:

• Explain the ScaledSquaring algorithm step by step (#57) (@benedict-96)
• Add opt-in optimizer step observer (#78) (@benedict-96)
• Allocate optimizer stand-ins on the argument's backend (#79) (@benedict-96)
• Normalise the retractions manual page to NFC (#80) (@michakraus)
• Start the line search example on a descent direction (#82) (@michakraus)
• CompatHelper: bump compat for SimpleSolvers to 0.14, (keep existing compat) (#83) (@github-actions[bot])
• Fix two GPU correctness bugs: ambient dot in ensure_descent!, Float64 literal in Stiefel metric (#84) (@michakraus)
• Fix copyto! and assign! on structured matrices to cross backends (#85) (@michakraus)
• Keep the LinearAlgebra contract in mul!, copyto! and matrix-vector products (#86) (@michakraus)
• Project an integer matrix into float(T), not into Float32 below Int64 (#87) (@michakraus)
• Reject Newton on a manifold and on a parameter set, with a message (#88) (@michakraus)
• Delete broadcast(f, ::Manifold), which claimed an invariant it did not hold (#89) (@michakraus)
• Add the symplectic Stiefel manifold and SR (#90) (@michakraus)
• Remove dead code and the runtime eval, and document the triangular adjoint aliasing (#91) (@michakraus)
• Separate the 52 ambiguous pairs between two of this package's own methods (#92) (@michakraus)
• Decide the element type of a backend-taking rand by one stated rule, and make host allocation one spelling (#93) (@michakraus)
• Refuse an unsupported element type at every backend-taking allocator (#94) (@michakraus)
• Orthonormalize with CholeskyQR2, so a manifold point can be drawn on a device (#95) (@michakraus)
• Multiply the triangulars and StiefelProjection without scalar indexing (#96) (@michakraus)
• Run check, rgrad and metric of a symplectic point on the point's backend (#97) (@michakraus)
• Settle a row vector times an owned matrix, and the transpose identities behind it (#98) (@michakraus)
• Import every name explicitly, bound the Test extra, and retire the stale comments (#99) (@michakraus)
• Point the lift testset's comment at where the retractions are covered (#100) (@michakraus)
• Group the Cayley retraction and the symplectic metric to drop their N^3 term (#101) (@michakraus)
• Export orthonormal_columns, so a downstream manifold layer can initialise on a device (#102) (@michakraus)
• Hold the retraction's buffers on the Optimizer, and gate the manifold step's allocations (#103) (@michakraus)
• Close the eight review findings from PR #103 (#104) (@michakraus)
• Let every optimizer state report the objective values it holds (#105) (@michakraus)
• Refuse a computation across two backends, by name (#107) (@michakraus)
• Refuse a structured matrix minus a plain matrix across two backends (#109) (@michakraus)
• Multiply a horizontal lift without scalar indexing (#110) (@michakraus)

Closed issues:

• Explain ScaledSquaring better (#56)

v0.7.0

GeometricOptimizers v0.7.0

Diff since v0.6.0

Breaking. A whole set of parameters is a NetworkParameters and never a bare NamedTuple. Everything else here follows from that.

ArrayNamedTuple was the cause, and the patches went with it

ParameterContainer{T} was Union{ArrayNamedTuple{T}, NetworkParameters{T}}, and its first member was NamedTuple{S,<:Tuple{Vararg{AbstractArray{T}}}} — an alias for Base.NamedTuple, not a type of its own. That is what made it a problem rather than merely a wide type, and it cost three separate things:

• A method on it was a method on Base.NamedTuple. That is why five of the eight sites of #16 had to move upstream to the packages owning their generics rather than be narrowed in place.
• It collided with GlobalSectionNamedTuple, an alias for NamedTuple too, and with the bare NamedTuple a nested section tree has to be written on. Every section-copy and parameter-copy pair in named_tuple_wrapper.jl overlapped on a shape neither method could claim. That was patched with four disambiguations; they are deleted here, because the overlap no longer exists.
• Its T meant two different things — for the NamedTuple half a guarantee that the set was flat and homogeneous, for the container a promotion over leaves at any depth.

Measured

Test.detect_ambiguities(GeometricOptimizers; recursive = true) on Julia 1.13:

	before	0.7.0
total reported	138	129
both methods owned here	35	26
the copyto! family	10	1

Two of the three "unreachable, documented in place" pairs are gone rather than documented — the empty NamedTuple and the mixed-element-type pair both needed a parameter set to be a NamedTuple. Prefer removing the cause to documenting the pair, and documenting it to adding a disambiguator: a method that exists only to satisfy a static checker is one somebody later has to reason about.

The one that remains is a GlobalSection{T,MT,Nothing} pair with no inhabitant this package's API can build. The other 26 are between this package's structured matrices and LinearAlgebra/ArrayLayouts methods on AbstractMatrix. None came from the de-piracy work.

Migrating

Wrap. NetworkParameters(ps) shares the leaf arrays, so an in-place solve still writes through to the arrays the caller holds:

ps = (Y = rand(StiefelManifold, 6, 3), W = randn(3, 4), b = zeros(6))
opt = Optimizer(NetworkParameters(ps), F; algorithm = Adam(Float64))

NetworkParameters is re-exported, the one exception to not re-exporting upstream's names: the wrap is this package's own entry condition now, and needing a second using to say so would make it look like a foreign concern.

_is_decayable has two methods, because it recurses through values itself and so meets a container's layers on the way down.

Type piracy is closed, as a property

test/aqua_tests.jl runs Aqua.test_piracies and asserts where each of the eight methods went. A list of sites goes stale — this package's own audit did, twice and in both directions — and a test does not.

• L2norm over a parameter set is NeuralNetworkParameters 0.3.0, in src/norms.jl.
• GradientAutodiff, GradientFunction and alloc_h for a parameter set, and GradientAutodiff for an AbstractMatrix, are SimpleSolvers 0.13.2.
• The two that could not follow are de-pirated in place: RiemannianGradient, a Gradient of this package's own that owns the projecting functor's signature, and the call-with-a-cache error, now on this package's own Hessian types — both halves, HessianBFGS and NoHessian.

Also fixed

docs/make.jl listed SimpleSolvers and GeometricMachineLearning as InterLinks inventories but not NeuralNetworkParameters, while RiemannianGradient's docstring @extrefs it. Documenter stopped at ExtCrossReferences and never reached RenderDocument. It hid because doctests run before cross-references, so a log filtered for "error" looked reassuring.

The three measurement harnesses timed on time(), which steps when the system adjusts it; they use the monotonic time_ns() now.

Requires

NeuralNetworkParameters 0.3.0, AbstractNeuralNetworks 0.8.0, SimpleSolvers 0.13.2. GeometricBase stays at 0.14.9 — the 0.14.10 that an earlier revision of this release needed is withdrawn, so that package is not in the chain.

0.6.1 was prepared and superseded before it reached the registry; its changes ship here.

Merged pull requests:

• Remove ArrayNamedTuple and ParameterContainer; a parameter set is a NetworkParameters (#74) (@michakraus)

Closed issues:

• Type piracy in l2norm, ParameterHandling.flatten, Gradient and outer! (#16)
• Performance: apply_toNT and _mul! allocate a fresh NamedTuple on every call (#23)

v0.6.0

GeometricOptimizers v0.6.0

Diff since v0.5.0

A NeuralNetworkParameters.NetworkParameters runs through the optimizer end to end. 0.5.0 made it a
member of OptimizerSolution{T}, which bound T at the eleven sites that take the element type from the
type of the solution and left everything else dispatching on ArrayNamedTuple — so a container got
several frames into a solve before failing. This finishes it: the elementwise primitives and the ~40
sites around them take a container, and the walks their bodies are written in are upstream's rather than
a local copy.

Breaking

• Julia 1.11 is the minimum, up from the 1.10 LTS. NeuralNetworkParameters' @generated walks need
  1.11 to be inlined. The CI matrix is 1.11 / 1.12 / 1.13 / nightly.
• Compat floors raised: GeometricBase = "0.14.9", the release that takes L2norm over any array,
  and NeuralNetworkParameters = "0.2.4", for the zipped foldparameters/foldstorage. Both are floors
  rather than widenings — the code calls into them unconditionally.
• l2norm(::AbstractMatrix) and l2norm(::AbstractFloat) are gone. They were the plainest type
  piracy in the package and they live upstream in GeometricBase 0.14.9 now.
• Two pirated Base.copyto! methods are gone: copyto!(::GlobalSectionNamedTuple, ::ParameterContainer)
  and copyto!(::NamedTuple, ::NetworkParameters). They are _copyto! methods at the same signatures, so
  dispatch resolves as it did; every caller here and in GeometricMachineLearning already went through
  _copyto!.
• Four outer!/_mul! methods are gone — outer! on a parameter set or an AbstractLieAlgHorMatrix,
  and _mul! on either. Both are always reached through the flat buffers now.
• update!(::BFGSState, ::Gradient, x, retraction) is gone. It had no caller in this package,
  GeometricMachineLearning or GMLDatasets; the live path is update!(state, opt, x).
• add! on a parameter set is deleted rather than narrowed. No caller anywhere, and
  GeometricMachineLearning deleted its own NamedTuple arm of AbstractNeuralNetworks.add! in 0.7 for
  the same reason.
• global_rep and apply_section rebuild in the shape of the parameters, not of the section. A
  container in gives a container back, where a section-driven walk gave a plain nested NamedTuple that
  no alias here covers.
• apply_section!(Y, λY::NamedTuple, Y₂) no longer takes Any in two of three slots. It is a
  ParameterSet signature now, which is what it always required.
• l2norm, solution_scale and _dot dispatch on ParameterSet, not ParameterContainer, so a
  nested plain NamedTuple is accepted where it used to raise — and, for l2norm, be splatted into
  L2norm(::NamedTuple, ::Matrix, …) and reported against a function the caller never named.
• Internal shape, for anyone reaching past the API: src/parameter_walks.jl and its
  _mapleaves/_mapleaves! are gone, _as_blocks and _as_walkable are gone, l2norm and
  solution_scale share one _sumsq_leaves, and _manifold_αmax bounds its first argument to a
  ParameterSet.

Fixed

• _dot of two horizontal lifts whose element types differ returned the ambient Frobenius product,
  at exactly twice the right value — silently, since no assertion on that path could tell the two
  answers apart. It reaches trial_slope, the quasi-Newton denominator δᵀγ and the predicted
  decrease, so a mixed-precision solve searched along a curve whose derivative was doubled. On
  St(6, 3): 5.504356027190567 before, 2.7521780135952834 after.
• The four folds over a parameter set are upstream's zipped fold, and the 26–71 s compile cliff on
  Julia 1.12 and 1.13 is gone. At 369 entries in one flat NamedTuple the sum over the four goes from
  109.85 s on 1.12 and 141.62 s on 1.13 to 3.82 s and 3.09 s, and the three version columns now agree.
  About a third of a second dearer on 1.11, which is the version that had nothing to gain.
• AdamCache's four-argument constructor: 88.23 s → 6.87 s on Julia 1.11.9 for a 16 × 24 nested
  container. It costs +4.7 s on 1.12 and +5.5 s on 1.13, so the worst cell across supported versions goes
  88 → 21 s and the spread collapses from 88/13/15 to 7/17/21.
• The flat buffers are no longer allocated per call. Every crossing a quasi-Newton method makes
  between the parameter shape and the flattened coordinates used to build a fresh vector.
• Two parameter sets are paired by key now, and the widths are checked — both in the generator, so
  free at run time. Crossed keys used to produce a number rather than an error.

The full entry, with the measurement tables and the open issues, is in
CHANGELOG.md.

Merged pull requests:

• Take a NetworkParameters through the optimizer (#68) (@michakraus)
• Stop building a flat vector on every crossing (#69) (@michakraus)
• Re-verify #68 and #69 against NeuralNetworkParameters 0.2.3 (#71) (@michakraus)
• The four folds over a parameter set are upstream's now (#72) (@michakraus)

Closed issues:

• The quadrature folds cost 26–71 s to compile on Julia 1.12 and 1.13 at 369 children in one branch (#70)

v0.5.0

GeometricOptimizers v0.5.0

Diff since v0.4.3

Breaking changes

ParameterHandling is no longer a dependency. The shim it existed for is deleted in full: the one-argument flatten(::ArrayNamedTuple{T}), the five methods on NamedTuple, Tuple, Tuple{}, Vector and AbstractMatrix/AbstractArray{,3} — type piracy that also shadowed methods ParameterHandling itself defines, since T<:AbstractFloat is narrower than its T<:Real — and the four methods on this package's own types, which src/parameter_protocol.jl supersedes. Flattening belongs to NeuralNetworkParameters now, and a ParameterLayout is a value rather than a closure:

v, layout = flatten(ps)
ps = unflatten(layout, v)

ForwardDiff is no longer a dependency. Its only use in src/ was a clause of the pirated flatten above, which existed so a round trip would not convert Duals back to the leaf's element type; NeuralNetworkParameters.unflatten keeps eltype(v) by construction. ForwardDiff still arrives transitively through SimpleSolvers, which is what actually differentiates a GradientAutodiff.

NeuralNetworkParameters is a hard dependency and NeuralNetworkParametersExt is gone. The extension's contents moved to src/parameter_protocol.jl unchanged, so the freeparameters/rebuild protocol is simply always present. Code reaching for the extension with Base.get_extension has to stop.

NeuralNetworkParameters 0.2 is required. The container carries the element type of its leaves as NetworkParameters{T}'s first type parameter, so a set is built from keys and values with NetworkParameters(NamedTuple{keys}(vals)) rather than NetworkParameters{keys}(vals).

apply_toNT is deleted. It was Base.map. It was unexported, and reached by qualified call from GeometricMachineLearning, whose 0.7.0 no longer makes it.

apply_toNT(f, a, b)   →   map(f, a, b)

Added

NetworkParameters{T} is a member of OptimizerSolution{T}, so a parameter set binds T in the where {T, VT<:OptimizerSolution{T}} clauses every cache and state constructor, both Optimizer constructors and the BFGSState update! methods are written with. This widens signatures rather than finishing the container swap: the elementwise primitives in src/optimizers/named_tuple_wrapper.jl still dispatch on ArrayNamedTuple.

changebackend for Manifold, VectorStorageMatrix and AbstractLieAlgHorMatrix, in a new AbstractNeuralNetworks weak-dependency extension, and GlobalSection(::NetworkParameters) — both previously carried by GeometricMachineLearning, which owned neither the function nor the types.

The full changelog is in CHANGELOG.md.

Merged pull requests:

• Drop ParameterHandling for the package that owns the parameters (#65) (@michakraus)

v0.4.3

GeometricOptimizers v0.4.3

Diff since v0.4.2

Merged pull requests:

• CompatHelper: bump compat for SimpleSolvers to 0.13, (keep existing compat) (#64) (@github-actions[bot])

v0.4.2

GeometricOptimizers v0.4.2

Diff since v0.4.1

A patch release: two additions and one fix, no breaking changes. Full changelog entry:
https://github.com/JuliaGNI/GeometricOptimizers.jl/blob/main/CHANGELOG.md

Added

• NativePade, a fifth AbstractExponentialAlgorithm and the second portable one. It evaluates
  𝔄 = φ₁ directly on the existing 2n×2n factor with the degree-6 diagonal Padé approximant q₆⁻¹p₆,
  whose coefficients are the [7/6] Padé approximant of exp rearranged, and replaces the dense solve
  a rational approximant needs with five fixed Newton–Schulz steps, so the whole path is reductions
  and matrix products. It exists as the independent portable implementation ScaledSquaring had
  been missing: both now run in the test suite on a JLArray, a backend that errors on scalar
  indexing. ScaledSquaring stays the default, and the measurements in docs/src/retractions.md say
  why. NativePade(θ) rejects θ > 1/2 — its five fixed iterations, unlike ScaledSquaring's series,
  stop being accurate above that. (#54)

• ScalarMomentAdam, a Stiefel-only baseline to compare Adam against. It is Cayley ADAM,
  Algorithm 2 of Li, Li and Todorovic, Efficient Riemannian Optimization on the Stiefel Manifold
  via the Cayley Transform (ICLR 2020), reproduced with this
  package's infrastructure. Its second moment is a scalar — one adaptive learning rate for the
  whole matrix instead of one per coordinate — which is the source's entire departure from ordinary
  Adam. Everything else is shared with Adam, which is what makes the two comparable on the same
  problem. Three deliberate departures from the source are recorded in the docstring: any of the
  package's retractions may replace the two-step Cayley approximation, which norm the second moment
  squares is the ambient_norm keyword, and the momentum is transported by the global section.
  ScalarMomentAdamState is public; the cache is not. (#51)

Fixed

• ScaledSquaring did not in fact run on a backend that forbids scalar indexing, which is the
  property it is the default for. 𝔄(A) opened with one(A), and Base.one(::AbstractMatrix)
  is similar, fill! and a scalar-indexed loop over the diagonal — so the documented claim that
  the default algorithm uses nothing but matrix products and norms was false one level down. The
  identity is now built with KernelAbstractions.zeros and the existing write_ones_kernel!, shared
  as unit_matrix by all five places that need one. (#54)

Documentation

• docs/src/retractions.md gains an allocation account and a Float32 forward-error table, both
  covering all five algorithms. The two rank them differently: ProjectedSkew is the best of the
  four on Float32 check by an order of magnitude and the worst on Float32 forward error at
  every norm, which is the trade it makes stated in both directions for the first time. (#54)

Merged pull requests:

• Add ScalarMomentAdam, a Stiefel-only Cayley ADAM baseline (#51) (@benedict-96)

v0.4.1

GeometricOptimizers v0.4.1

Diff since v0.4.0

Merged pull requests:

• Restore theme-specific documentation figures (#59) (@benedict-96)
• Give the structured matrices a NeuralNetworkParameters protocol (#60) (@michakraus)

Closed issues:

• Evaluate a native Padé-based low-rank exponential algorithm (#58)

v0.4.0

GeometricOptimizers v0.4.0

Diff since v0.3.1

Changed (breaking)

The manifold, section and retraction interface is now exported. These names were internal, so
downstream code reached them GeometricOptimizers.-qualified or kept copies of its own:

geometry	metric, check, Ω (rgrad was already exported)
matrices	AbstractTriangular, StiefelProjection, AbstractLieAlgHorMatrix
sections	global_section, apply_section, apply_section!, update_section!
retractions	AbstractRetraction, Geodesic, Cayley, geodesic, cayley, retraction
optimizer	OptimizerMethod, OptimizerSolution

This is the breaking part of the release: a name that is now exported can collide with a downstream
one, and using two packages that both export it is an UndefVarError on that name until one of
them qualifies or aliases it. Twelve of these did collide with GeometricMachineLearning, which is
why they are exported — GML kept near-verbatim copies of eleven types defined here because the
interface it needed was internal, and GeometricMachineLearning 0.5 deletes about 2500 lines of
those copies against this release.

The optimizer caches stay internal, for every method alike: GradientCache, MomentumCache,
AdamCache, BFGSCache, DFPCache, NewtonOptimizerCache and the OptimizerCache supertype. They
are solver_step! scratch.

Added

• SymmetricMatrix, LowerTriangular and UpperTriangular can be optimizer parameters. The
  methods an optimizer needs did not exist for them: similar, fill!, the elementwise _add!,
  _rac!, _square!, _div!, _rmul!, _difference!, then l2norm,
  ParameterHandling.flatten and update_section! on a GlobalSection over one of them. The
  generic array methods either broadcast — and three of these four types have no setindex! to
  broadcast through — or reshape, and n(n±1)/2 numbers do not reshape back to n × n.

  They are written once over a new VectorStorageMatrix alias for the four types that keep their
  free parameters in one vector, which is also what the SkewSymMatrix-specific methods that
  existed before collapse into.

• Docstrings for retraction and update_section!, which had none.

• Tests for the triangular types, which had none, and for the primitives above — including a run of
  each of the three first-order methods over each of the four types. The suite goes from 7934 to
  8347 tests.

Fixed

• Four tests that were not testing what they claimed: three functions in
  test/special_matrices/skew_symmetric.jl were defined and never called (one carrying a
  SkeySymMatrix typo), stiefel_global_section built a GrassmannManifold so the Stiefel global
  section went untested, and stiefel_proj's element type defaulted to Flaot32.

• scripts/retraction_accuracy.jl loads again; 0.3.0's rename of _BFGS/_DFP to BFGS/DFP
  missed it.

Documentation

The manifold, special-matrix and optimizer chapters move here from GeometricMachineLearning —
thirteen pages, documenting types that live here — with their TikZ figures as sources, compiled in
the documentation job.

See CHANGELOG.md for the
full entry.

Merged pull requests:

• Repair scripts/retraction_accuracy.jl, which 0.3.0 renamed out from under (#49) (@michakraus)
• Make the geometry public API, and take in its documentation (#50) (@michakraus)