Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

write >> as infix notation for OptimiserChain #139

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

write >> as infix notation for OptimiserChain #139

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
99ae33d
write => for OptimiserChain
mcabbott Nov 27, 2022
4a55975
change from => to >>
mcabbott Sep 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions src/rules.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -636,10 +636,12 @@ function _norm(dx::Broadcast.Broadcasted, p::Real)
end

"""
OptimiserChain(opts...)
OptimiserChain(o1, o2, o34...)
o1 >> o2 >> o3

Compose a sequence of optimisers so that each `opt` in `opts`
Compose a sequence of optimisers so that each `opt` in `(o1, o2, o34...)`
updates the gradient, in the order specified.
May be entered using the `>>` operator with several `AbstractRule`s.

With an empty sequence, `OptimiserChain()` is the identity,
so `update!` will subtract the full gradient from the parameters.
Expand All @@ -648,12 +650,13 @@ This is equivalent to `Descent(1)`.
# Example

```jldoctest
julia> o = OptimiserChain(ClipGrad(1.0), Descent(0.1));
julia> o = ClipGrad(1.0) >> Descent(0.1)
OptimiserChain(ClipGrad(1.0), Descent(0.1))

julia> m = (zeros(3),);

julia> s = Optimisers.setup(o, m)
(Leaf(OptimiserChain(ClipGrad(1.0), Descent(0.1)), (nothing, nothing)),)
(Leaf(ClipGrad(1.0) => Descent(0.1), (nothing, nothing)),)

julia> Optimisers.update(s, m, ([0.3, 1, 7],))[2] # clips before discounting
([-0.03, -0.1, -0.1],)
Expand All @@ -664,6 +667,11 @@ struct OptimiserChain{O<:Tuple} <: AbstractRule
end
OptimiserChain(opts...) = OptimiserChain(opts)

@doc @doc(OptimiserChain)
Base.:(>>)(a::AbstractRule, b::AbstractRule) = OptimiserChain(a, b)
Base.:(>>)(a::AbstractRule, bc::OptimiserChain) = OptimiserChain(a, bc.opts...)
Base.:(>>)(ab::OptimiserChain, c::AbstractRule) = OptimiserChain(ab.opts..., c)

@functor OptimiserChain

init(o::OptimiserChain, x::AbstractArray) = map(opt -> init(opt, x), o.opts)
Expand All @@ -679,7 +687,14 @@ function apply!(o::OptimiserChain, states, x, dx, dxs...)
end
end

function Base.show(io::IO, c::OptimiserChain)
function Base.show(io::IO, c::OptimiserChain) # compact show
if length(c.opts) > 1
join(io, c.opts, " >> ")
else
show(io, MIME"text/plain"(), c)
end
end
function Base.show(io::IO, ::MIME"text/plain", c::OptimiserChain)
print(io, "OptimiserChain(")
join(io, c.opts, ", ")
print(io, ")")
Expand Down
Loading