There is no interation between no_grad, enable_grad, set_grad_enabled

[Yura52]

📚 Documentation

torch==1.5.0

The problem is the same for all three functions. For example, let's consider no_grad.

The documentation says: "This mode has no effect when using enable_grad context manager". This is not true:

import torch
x = torch.tensor([1.0], requires_grad=True)
with torch.enable_grad():
    with torch.no_grad():
        y = x * 2
assert not y.requires_grad

And I mean that the documentation should be fixed, not the behavior of torch 😄
These functions just set grad enabled/disabled in __enter__ (or in the constructor in the case of set_grad_enabled) and set the previous state in __exit__. This is very intuitive. IMHO, it will be confusing if, for example, the behavior of no_grad changes depending on some other context.

cc @ezyang @gchanan @zou3519 @ssnl @albanD @gqchen @jlin27

[colesbury]

This was changed in #23310 apparently because of #19189.

The complaint in #19189 was legitimate -- the original terminology was confusing. But the PR #23310 confuses things by also adding erroneous claims:

https://github.com/pytorch/pytorch/pull/23310/files#diff-a1d32fe3effc8d03f2cf3e8af9216de9R106-R107

https://github.com/pytorch/pytorch/pull/23310/files#diff-a1d32fe3effc8d03f2cf3e8af9216de9R15

We should have caught that in the PR review. Oops.

[colesbury]

I'm bumping this to high priority considering @ezyang marked #19189 high-pri. The current documentation is incorrect. We should remove these two statements:

This mode has no effect when using :class:`~enable_grad` context manager .

    When using :class:`~enable_grad` context manager, :class:`~set_grad_enabled(False)`
    has no effect.

[vfdev-5]

@colesbury just to confirm, this issue is about documentation only (no code change required), right ?

[colesbury]

@vfdev-5 yes, documentation only

[peterbell10]

@vfdev-5 friendly reminder to assign yourself to issues you're working on.