Fix sigmoid grad

[t-ae]

I tried the code below on colab.

import TensorFlow

@differentiable
public func mysigmoid<T: TensorFlowFloatingPoint>(_ x: Tensor<T>) -> Tensor<T> {
    return 1 / (1 + exp(-x))
}

let dsqrt: (Tensor<Float>)->Tensor<Float> = gradient(of: sqrt)
let dsigmoid: (Tensor<Float>)->Tensor<Float> = gradient(of: sigmoid)
let dmysigmoid: (Tensor<Float>)->Tensor<Float> = gradient(of: mysigmoid)

// Check sigmoid and mysigmoid are same
let ten: Tensor<Float> = [-1, -0.5, 0, 0.5, 1]
print(sigmoid(ten))
print(mysigmoid(ten))

// The gradient of sqrt where x=0.5 (= 0.7071...)
print(dsqrt(Tensor<Float>(0.5))) // 0.36787945

// The gradient of mysigmoid where x=0.9 (=0.2055...)
print(dmysigmoid(Tensor<Float>(0.9))) // 0.2055003

// Expects 0.2055...
print(dsigmoid(Tensor<Float>(0.9))) // 0.09000002

// This returns 0.2055..
print(dsigmoid(sigmoid(Tensor<Float>(0.9)))) // 0.20550032

It looks dsigmoid returns wrong value.

sigmoid and its gradient is defined here.

swift-apis/Sources/TensorFlow/Operators/Math.swift

Lines 962 to 975 in 323b564

	/// Returns the sigmoid of the specified tensor element-wise.
	/// Specifically, computes `1 / (1 + exp(-x))`.
	@inlinable
	@differentiable(vjp: _vjpSigmoid)
	public func sigmoid<T: TensorFlowFloatingPoint>(_ x: Tensor<T>) -> Tensor<T> {
	Raw.sigmoid(x)
	}
	@inlinable
	internal func _vjpSigmoid<T: TensorFlowFloatingPoint>(
	_ x: Tensor<T>
	) -> (Tensor<T>, (Tensor<T>) -> Tensor<T>) {
	(sigmoid(x), { v in Raw.sigmoidGrad(x, dy: v) })
	}

And Raw.sigmoidGrad is here.
https://raw.githubusercontent.com/tensorflow/swift-apis/323b5640c48822340f2f5410d6bff51972fc9f45/Sources/TensorFlow/Bindings/RawOpsGenerated.swift

/// Computes the gradient of the sigmoid of `x` wrt its input.
///
/// Specifically, `grad = dy * y * (1 - y)`, where `y = sigmoid(x)`, and
/// `dy` is the corresponding input gradient.
@inlinable @inline(__always)
public static func sigmoidGrad<T: FloatingPoint & TensorFlowScalar>(
    _ y: Tensor<T>,
    dy: Tensor<T>
)

_vjpSigmoid passes x but what sigmoidGrad requires is y.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[t-ae]

By the way, I added test code but couldn't test in my environment. (but tried equivalent code on colab.)
When I run swift build, it fails with the message below.

Assertion failed: (isa<GenericTypeParamDecl>(D) && "unexpected decl kind"), function isDeclXRef, file /Users/danielzheng/swift-tf/swift/lib/Serialization/Serialization.cpp, line 2043.

(It's odd message, since I'm not danielzheng, of course)

I want someone to test my code before merge.

[t-ae]

@googlebot I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[eaplatanios]

Thanks for catching this @t-ae!

[eaplatanios]

@t-ae some tests fail with this error:

Test Case 'LayerTests.testSequential' started at 2019-07-10 03:42:15.515
/swift-apis/Tests/TensorFlowTests/LayerTests.swift:53: error: LayerTests.testSequential : XCTAssertEqual failed: ("0.24223351") is not equal to ("0.25301588") +/- ("0.0001") -
/swift-apis/Tests/TensorFlowTests/LayerTests.swift:53: error: LayerTests.testSequential : XCTAssertEqual failed: ("0.20923203") is not equal to ("0.21743035") +/- ("0.0001") -
/swift-apis/Tests/TensorFlowTests/LayerTests.swift:53: error: LayerTests.testSequential : XCTAssertEqual failed: ("0.32405508") is not equal to ("0.32044548") +/- ("0.0001") -
/swift-apis/Tests/TensorFlowTests/LayerTests.swift:53: error: LayerTests.testSequential : XCTAssertEqual failed: ("0.28849685") is not equal to ("0.2807928") +/- ("0.0001") -

Could you please update them appropriately?

[t-ae]

@eaplatanios
That test uses sigmoid and its gradient so result is changed.

It's you who added this test. How did you get these expected values?
https://github.com/tensorflow/swift-apis/blame/5b4300e9746204cbde006f8f7af59964d0e88a9c/Tests/TensorFlowTests/LayerTests.swift#L28-L57

I suspect these values were got from this code itself.
It's easy to simply update values to match results. But it's not valid for testing IMO.

[t-ae]

As I wrote above, I can't build in my environment currently. I'll try that at first.
I'm glad if someone update test while I'm trying to build.

[t-ae]

OK, I managed to build.

I'm not sure what is the main purpose of this test.
(Building Sequential model? its differentiability?)

Anyway, it looks the result values are not what count.
So I changed the test to simply check if loss is descending.

[eaplatanios]

Yes a lot of the tests are not super well organized/justified right now. Usually we try to make them match some existing working implementation (e.g., using Python). This looks good to me now. Thanks for updating it!