Fix elu backward operation for negative alpha

[H-Huang]

Fixes #47671

Test:

x = torch.tensor([-2, -1, 0, 1, 2], dtype=torch.float32, requires_grad=True)
y = torch.nn.functional.elu_(x.clone(), alpha=-2)
grads = torch.tensor(torch.ones_like(y))
y.backward(grads)

RuntimeError: In-place elu backward calculation is triggered with a negative slope which is not supported. 
This is caused by calling in-place forward function with a negative slope, please call out-of-place 
version instead.

[albanD]

Hi,

Thanks for sending a patch.
The forward computation is correct right, even when alpha is negative?
And the backward would work as well, as long as the forward was not inplace if we use the input properly?
So I think maybe doing something similar to leaky relu here would allow the users to keep using negative alpha. As long as they don't use it inplace.
You can update the backward formula to use the boolean flag set in the formula to only error out in the case that is not supported and allow all the others to work!

[H-Huang]

Hi @albanD, thank you for the feedback. According to the example in #47671, we are hitting an issue when alpha is negative for the backward calculation even when the forward was out-of-place. So I updated the PR to check if the alpha is negative OR if the forward was done in-place, and if so then prevent backward calculation. Is this understanding correct?

In the leaky relu case it looks like it checks if the slope is negative AND the forward was in-place, which is slightly different.

[albanD]

we are hitting an issue when alpha is negative for the backward calculation even when the forward was out-of-place

I think this only happens because the formula here uses result instead of self. If you are using self there, that would make the out of place version work for negative alpha.
But as mentioned in the other PR, this would make the inplace (that uses the same formula from derivatives.yaml by default if none is provided) use extra memory which we don't want. Hence the explicit inplace formula that sets the proper flag telling that it is passing result instead of self (and thus allowing to raise a nice error in this one case that we cannot support).

[H-Huang]

I see, thank you for the helpful explanations @albanD! Updated the PR accordingly.

[facebook-github-bot]

@H-Huang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@H-Huang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[albanD]

Let me know when CI is green and this is ready for a final review :)

[H-Huang]

@albanD Can you take a look whenever you get the chance?

[albanD]

LGTM !

[facebook-github-bot]

@H-Huang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@H-Huang merged this pull request in ec51b67.