Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tangents are not being propagated through the relu and sigmoid activations. #16

Closed
mrakgr opened this issue Dec 24, 2015 · 2 comments
Closed
Labels

Comments

@mrakgr
Copy link
Contributor

mrakgr commented Dec 24, 2015

static member ReLU (a:D) =
    let inline ff(a) = max 0.f a
    let inline fd(a) = D.ReLU(a)
    let inline df(cp, ap, at) = (1.f + D.Sign(ap)) / 2.f
    let inline r(a) = ReLU_D(a)
    D.Op_D_D (a, ff, fd, df, r)

static member Sigmoid (a:D) =
    let inline ff(a) = 1.f / (1.f + exp -a)
    let inline fd(a) = D.Sigmoid(a)
    let inline df(cp:D, ap, at) = cp * (1.f - cp)
    let inline r(a) = Sigmoid_D(a)
    D.Op_D_D (a, ff, fd, df, r)

In the df functions there should be an at in there somewhere. The DV and DM versions are similarly affected.

@gbaydin
Copy link
Member

gbaydin commented Dec 24, 2015

Yes, this is a bug affecting the forward AD of ReLU and Sigmoid. Thank you very much for reporting it.

It is now fixed in the repository.

@gbaydin gbaydin added the bug label Dec 24, 2015
@gbaydin gbaydin closed this as completed Dec 24, 2015
@gbaydin
Copy link
Member

gbaydin commented Dec 25, 2015

This is fixed in version 0.7.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants