v0.3.3

Optimisers v0.3.3

Diff since v0.3.2

Merged pull requests:

• add missing SignDecay doc reference (#168) (@CarloLucibello)
• fix broken doc links (#170) (@CarloLucibello)
• add trainables (#171) (@CarloLucibello)
• fix broken documentation (#172) (@CarloLucibello)
• add path option to trainables (#174) (@CarloLucibello)

Closed issues:

• Documenter CI is failing (#169)

v0.3.2

Optimisers v0.3.2

Diff since v0.3.1

Merged pull requests:

• WeightDecay for L1 norm (#159) (@mcabbott)
• Add all-keyword constructors, much like @kwdef (#160) (@mcabbott)
• CompatHelper: add new compat entry for Statistics at version 1, (keep existing compat) (#164) (@github-actions[bot])
• Don't load Yota at all (#166) (@mcabbott)

v0.3.1

Optimisers v0.3.1

Diff since v0.3.0

Closed issues:

• Error in update! for Metal arrays and Adam optimiser (#150)

Merged pull requests:

• Fix Float64 beta in Adam etc. (#158) (@mcabbott)

v0.3.0

Optimisers v0.3.0

The major change is #151, which removes type parameterisation from the structs. This should not break straightforward user code, but may break break loading via BSON etc. It also adds errors on negative learning rates, and will in some cases change the default regulator from eps(Float32) to 1e-8.

Diff since v0.2.20

Closed issues:

• Adam(0) fails (#119)
• Error in update! for Metal arrays and Adam optimiser (#150)

Merged pull requests:

• Use eltype(x) everywhere, ignore typeof(η) (#151) (@mcabbott)

v0.2.20

Optimisers v0.2.20

Diff since v0.2.19

Closed issues:

• Implement Lion, up to 5x faster than Adam, and more accurate (#156)

Merged pull requests:

• Add Lion to docs (#157) (@ToucheSir)

v0.2.19

Optimisers v0.2.19

Diff since v0.2.18

Closed issues:

• OptimiserChain(..., ClipNorm) fails on GPU (#127)

Merged pull requests:

• Don't use state anywhere for the whole state tree (#136) (@mcabbott)
• Update TagBot config (#142) (@ToucheSir)
• Make ClipNorm work on GPU Broadcasted (#144) (@mcabbott)

v0.2.18

Optimisers v0.2.18

Diff since v0.2.17

Closed issues:

• Interface for gradient accumulation (#130)
• Optimisers.update fails with gradient of type CUDA.CUSPARSE.CuSparseMatrixCSC (#141)

Merged pull requests:

• Rule for gradient accumulation (#137) (@CarloLucibello)

v0.2.17

Optimisers v0.2.17

Diff since v0.2.16

Merged pull requests:

• fix: rm Functors@0.3 (#134) (@ven-k)

v0.2.16

Optimisers v0.2.16

Diff since v0.2.15

Closed issues:

• use structs instead of functions for walks in destructure (#124)
• Documentation error (#131)

Merged pull requests:

• fix anonymous walk deprecation (#125) (@CarloLucibello)
• fix typo (#132) (@bicycle1885)

v0.2.15

Optimisers v0.2.15

Diff since v0.2.14

Merged pull requests:

• update: actions node 12 => node 16 (#126) (@skyleaworlder)
• Discourage use of trainable (#128) (@mcabbott)
• Add implementation of Lion optimiser (#129) (@mashu)