Adding GlobalMaxPooling 1D, 2D, 3D

[Shashi456]

I've added the max operation to operators, since it wasnt defined earlier.

[dan-zheng]

Thank you! Could you please add some tests to Tests/DeepLearningTests/LayerTests.swift?

[rxwei]

Very nice. Thank you! As @dan-zheng said, tests should be added.

[Shashi456]

@dan-zheng I'm working on the test cases now, I will have a seperate PR for them, there are a lot of layers which do not have test cases and I will make a list of the same in #77, it would be easier to track them that way.

[Shashi456]

@rxwei can this be merged?

[rxwei]

Could you pull from master? I wanted to trigger a rebuild.

[Shashi456]

@rxwei the docker summary fails because it says the max function is not differentiable, is there any way correct that?

[tanmayb123]

We could either add a VJP for the max op to the standard library, or just use the actual "max pooling" operation inside the global max pooling layers (set the pool size to the width & height of the input).

[Shashi456]

@tanmayb123 I was trying to do that when i initially added the max function to the batch normalized vjp, i just put it in the wrong place. I think we should rather put in the max function because it is actually used in multiple places, might come in as a handy operation.

[Shashi456]

@rxwei could you tell me how i could go about defining a vjp for the max function?
I think the vjp is the way to go since we'd need a max function for hinge loss as well.

[rxwei]

I'm sorry for the delayed response! Recently the team has been busy with releasing v0.3 and getting some important core pieces like RNNs into the library. I'll definitely help you with defining derivatives for min and max later this week!

[rxwei]

In the meantime, it would be useful to explore our custom differentiation tutorial, and see how other derivatives are defined, e.g. mean(alongAxes:). Feel free to post any questions you have.

[Shashi456]

@rxwei do you mind triggering a build on this? Thanks to @eaplatanios for adding the max and min vjps. With this, we are done with all the pooling layers.

[Shashi456]

@rxwei could you take a quick glance and trigger the build once more? This does pass locally now.

Also is there any way to keep updating the toolchain directly, instead of continuously downloading nightlies xD.

[rxwei]

Also is there any way to keep updating the toolchain directly, instead of continuously downloading nightlies xD.

You can always build a toolchain from the tensorflow branch on apple/swift using the utils/build-toolchain-tensorflow script. However, it often takes quite a while.

[Shashi456]

@eaplatanios sorry to bother you with this. But could you tell me if the max function call i wrote for this is wrong? The error that pops is function not differentiable

[eaplatanios]

@Shashi456 Could you post the full error trace? I can clone and take a look in a bit.

[Shashi456]

[17/36] Compiling TensorFlow Pooling.swift
/swift-apis/Sources/TensorFlow/Layers/Pooling.swift:375:22: error: expression is not differentiable
        return input.max(squeezingAxes: [1, 2])
                     ^
/swift-apis/Sources/TensorFlow/Layers/Pooling.swift:375:22: note: cannot differentiate functions that have not been marked '@differentiable' and that are defined in other files
        return input.max(squeezingAxes: [1, 2])
                     ^
/swift-apis/Sources/TensorFlow/Layers/Pooling.swift:359:22: error: expression is not differentiable
        return input.max(squeezingAxes: 1)
                     ^
/swift-apis/Sources/TensorFlow/Layers/Pooling.swift:359:22: note: cannot differentiate functions that have not been marked '@differentiable' and that are defined in other files
        return input.max(squeezingAxes: 1)
                     ^
/swift-apis/Sources/TensorFlow/Layers/Pooling.swift:391:22: error: expression is not differentiable
        return input.max(squeezingAxes: [1, 2, 3])
                     ^
/swift-apis/Sources/TensorFlow/Layers/Pooling.swift:391:22: note: cannot differentiate functions that have not been marked '@differentiable' and that are defined in other files
return input.max(squeezingAxes: [1, 2, 3])

[eaplatanios]

I see. The Tensor.max(squeezingAxes:) function is not currently differentiable when the axes are int arrays. I'll make a PR to fix that now.

[eaplatanios]

Done in #159.

[rxwei]

#159 has been merged. Could you do a pull?

[Shashi456]

@rxwei done. pretty happy that finally done with pooling layers xD

[Shashi456]

@rxwei I'll take care of both the PRs, in a while. Sorry!

[Shashi456]

@rxwei built and tested with nightly. This PR passes as is. The build now takes substantially more time though somehow xD.

I'm not sure about the test since the linux error you've mentioned popped up, but i rectified the error you mentioned about GlobalMaxPooling1D.

Edit: Tested this by commenting out the Linux errors, The test added by this PR pass.

I think this is good to go.

[Shashi456]

@rxwei could you trigger a build on this?

[Shashi456]

@rxwei The errors aren't from these additions. I think this can be merged,

[Shashi456]

@rxwei I think this is ready to be merged.