Move GradMode / AutoGradMode / NoGradGuard to ATen core

[yf225]

After the Variable/Tensor merge, code paths in ATen need to be able to check whether a tensor requires gradient, and throw errors in places where a requires_grad=true tensor is not allowed (such as https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/Utils.h#L76-L78 and https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/SparseTensorImpl.cpp#L86). Since the GradMode thread-local variable controls whether a tensor should accumulate gradients, we need to be able to check this variable from ATen when we determine whether a tensor requires gradient, hence the PR to move GradMode / AutoGradMode / NoGradGuard to ATen.

Note that we intentionally don't merge at::GradMode and at::NonVariableTypeMode, with the following reasoning:
Semantically, at::GradMode and at::NonVariableTypeMode actually mean different things: at::GradMode controls whether a tensor should accumulate gradients, and at::NonVariableTypeMode controls whether a Variable should be treated as a non-Variable tensor in type dispatches. There are places whether we don't want the tensor to accumulate gradients, but still want the Variable to be treated as a Variable. Here is one example:

#  torch/tensor.py
with torch.no_grad():
   ...
   new_tensor = self.new()    # `at::GradMode` is false at this point
   ...

// tools/autograd/templates/python_variable_methods.cpp
static PyObject * THPVariable_new(PyObject* self, PyObject* args, PyObject* kwargs)
{
  ...
  // if we merge `at::GradMode` and `at::NonVariableTypeMode`, since `at::GradMode` is false and `self_.type()` checks `at::GradMode` to decide whether to return non-Variable type, it will return a non-Variable type here, which is not what we want (and throws a "Tensor that was converted to Variable was not actually a Variable" error)
  return THPVariable_Wrap(torch::utils::legacy_tensor_new(self_.type(), args, kwargs));
  ...
}

For the above reason, we cannot merge at::GradMode and at::NonVariableTypeMode, as they have different purposes.

[yf225]

Note from offline discussion with @gchanan : we should look into how to avoid moving GradMode to ATen, and change places where this move is expected.

[ezyang]

Note that we intentionally don't merge at::GradMode and at::NonVariableTypeMode, with the following reasoning

This would make a really good Note in the source code

[ezyang]

okey dokey

[gchanan]

the note on the offline discussion says we don't actually need this, right?

[yf225]

#22473 requires that we check torch::autograd::GradMode::is_enabled() in Caffe2 tensor's enforce_invariants() method in caffe2/core/tensor.cc. Since caffe2/core/tensor.cc is in Caffe2 core, @ljk53 suggested that we should move torch::autograd::GradMode from torch/csrc/autograd to ATen core, so that core code doesn't need to depend on non-core code (i.e. torch/csrc/autograd), which can prevent build structure headache.

[facebook-github-bot]

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@yf225 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@yf225 merged this pull request in 221af09.