torch.clamp kills gradients at the border

[SevenBlocks]

Issue description

torch.clamp() should only affect gradients for values outside the min and max range, but it also appears to affect values equal to the min or max. I.e. clamping the value 0 to min=0, max=1 should have no effect on the gradient for that value, but it does--the gradient is being set to 0.

Code example

a=Variable(torch.tensor([0.0]), requires_grad=True)

# Normal result, gradient of a with respect to a is 1
> torch.autograd.grad(a, a)
(tensor([ 1.]),)

# Incorrect result
> torch.autograd.grad(torch.clamp(a, 0, 1), a)
(tensor([ 0.]),)

# Setting the min just below 0 also produces the correct result
> torch.autograd.grad(torch.clamp(a, -0.1, 1), a)
(tensor([ 1.]),)

System Info

Collecting environment information...
PyTorch version: 0.4.0
Is debug build: No
CUDA used to build PyTorch: 8.0.61

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: GeForce GTX 980 Ti
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 384.111
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.0.5
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.1.5
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6.0.21
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a

Versions of relevant libraries:
[pip3] numpy (1.14.1)
[pip3] torch (0.4.0)
[pip3] torchvision (0.2.0)
[conda] Could not collect

• PyTorch or Caffe2:
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• OS:
• PyTorch version:
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• GCC version (if compiling from source):
• CMake version:
• Versions of any other relevant libraries:

[zou3519]

clamp is not differentiable everywhere. It is differentiable on (min, max), (-infty, min), and (max, infty). Because of that, I would expect it to not have gradients at min or max: they're being set to zero here, which is good.

[ssnl]

As @zou3519 said, gradient is not properly defined at min and max.

[SevenBlocks]

Clamp should be differentiable for points that were equal to the min or max before any clamping occurred. It should only zero the gradients for values that were outside the range. Is there a reason it can't keep track of which values were outside the range? Theano calculates these gradients correctly.

[ssnl]

@SevenBlocks There is not such thing as calculating gradients correctly because mathematically the gradient is not defined at those two points. There are only subgradients, which can be anywhere in [0, 1].

[SevenBlocks]

The clamp function is f(x) =
x, if min <= x <= max
min, if x < min
max, if x > max

df/dx is:
1, if min <= x <= max
0, otherwise

I don't see how it's not well defined.

[zou3519]

What's the derivative of f(x) at x = min?

Taking the definition of a derivative here: http://tutorial.math.lamar.edu/Classes/CalcI/DefnOfDerivative.aspx , the derivative is a limit. This limit exists if the limit from the right equals the limit from the left.

For x = min, the limit from the left ("slope of the line immediately to the left of x") is 0, while the limit from the right ("slope of the line to the right of x") is 1. By the definition of limit, the limit at this point, x = min doesn't exist. So f(x) is not differentiable at x = min.

[zou3519]

On the other hand, @SevenBlocks, I do understand your feature request. I'm not sure about the following though:

1. What the use cases for this are
2. If we say the derivative is 1, if anything bad will happen

[SevenBlocks]

I see, you're right. Intuitively, though, it seems like 1 or 0.5 might be better than 0.

1. It makes it difficult to reproduce/rewrite Theano code in pytorch if the gradients are calculated differently.
2. I'm not sure, but I don't see how changing it would be worse.

[zou3519]

Our users might rely on the edge case that the gradient does not exist. At any rate, you can define a custom autograd function in Python that has its own forward and backward. The forward can use clamp, while the backward should look something like:

grad_output * (input >= min).type_as(grad_output) * (input <= max).type_as(grad_output)

and then use it in place of clamp.

[ssnl]

Let’s also bear in mind that this is just a single point in the entire float range. The likelihood that your training is going to land on this point (okay two points) is very very very small. Even it does, it will mostly be just one time and the probablity that this causes the result to differ greatly is virtually zero. Autograd frameworks differ on points like this all the time. Taking the mathematical view (i.e. pretending computers have continuous numbers), they are all correct in that they all calculate subgradients. And probably more important to you, they are equal almost everywhere on the real line, which means that if you pick a point in R, the probablity that you land in such points is 0.

…

On Fri, Apr 27, 2018 at 3:31 AM Richard Zou ***@***.***> wrote: Our users might rely on the edge case that the gradient does not exist. At any rate, you can define a custom autograd function in Python that has its own forward and backward. The forward can use clamp, while the backward should look something like: grad_output * (input >= min).type_as(grad_output) * (input <= max).type_as(grad_output) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#7002 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFaWZcWhzJll6oz0sq_EWrf0HdjRE0MGks5tsiB8gaJpZM4TmW_u> .

[SevenBlocks]

@ssnl if all values were random the probability would be small, but that's not the case generally. The Equilibrium propagation algorithm by Bengio et al. initializes states to zero which are clamped to 0 and 1 and updated based on gradients of an energy function. It seems to rely on the gradient being 1 when states are 0 to update properly. I will try the suggestion by @zou3519. Thanks.

[fmassa]

@ssnl @zou3519 while I agree that the gradient at the boundaries is not well defined, I think that it might be worth considering letting the gradient at the boundaries be 1 instead of zero. This seems to also follow the convention of TF, and I don't really think that people used to rely on this behavior of the derivative of clamp, so it wouldn't be a breaking change

[zou3519]

@fmassa do you have example code for tf that shows its behavior on this? (Sorry, I'm not very good with tf)