Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nan gradient for elu and selu. #758

Closed
racinmat opened this issue Aug 7, 2020 · 3 comments
Closed

Nan gradient for elu and selu. #758

racinmat opened this issue Aug 7, 2020 · 3 comments

Comments

@racinmat
Copy link
Contributor

racinmat commented Aug 7, 2020

julia> using Zygote
julia> gradient(x -> sum(Zygote.selu.(x)), [1_000.0])
([1.0507009873554805],)
julia> gradient(x -> Zygote.selu(x), 1_000.0)
(NaN,)
julia> gradient(x -> sum(Zygote.elu.(x)), [1_000.0])
([NaN],)
julia> gradient(x -> Zygote.elu(x), 1_000.0)
(NaN,)

the broadcasted version for Selu works because of https://github.com/FluxML/Zygote.jl/blob/master/src/lib/nnlib.jl#L17
but I assume non-broadcasted version should be added and the elu derivation should also be added, right?

@DhairyaLGandhi
Copy link
Member

+1 for this

racinmat added a commit to racinmat/Zygote.jl that referenced this issue Aug 13, 2020
@racinmat
Copy link
Contributor Author

Created PR but it would be great if someone reviewed if it's correct.

bors bot added a commit that referenced this issue Aug 18, 2020
765: fix for #758 r=DhairyaLGandhi a=racinmat

Added few tests, defined adjoint for scalar selu and elu.
Fixes #578 
I hope this is right, I'm still not sure if I fully got all the tricks around adjoints.

Co-authored-by: Matěj Račinský <matej.racinsky@avast.com>
@racinmat
Copy link
Contributor Author

Closed by #765

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants