[dynamo][TensorVariable] Support "if param.grad_fn" usecase

[anijain2305]

Stack from ghstack (oldest at bottom):

• [dynamo] Initial exception handling support #126923
• [dynamo] Support __getitem__ on NNModuleVariable __dict__ #126956
• -> [dynamo][TensorVariable] Support "if param.grad_fn" usecase #126960
• [dynamo] Support __contains__ on obj.__dict__ #126922

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126960

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 365e6c8 with merge base d62b025 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2) (gh) (similar failure)
  test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_Transformer_cuda_float64

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#126884)
  eca_halonext26ts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[anijain2305]

@jansel @bdhirsh With this PR, this simple repro fails

import torch


def fn():
    p = torch.nn.Parameter(torch.empty(4, 4))
    with torch.no_grad():
        # breakpoint()
        return p.add_(1)

print(fn())

opt_fn = torch.compile(fn, backend="eager")
print(opt_fn())

with error message

  File "/data/users/anijain/pytorch2/torch/_dynamo/variables/builder.py", line 1693, in wrap_fx_proxy
    return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/anijain/pytorch2/torch/_dynamo/variables/builder.py", line 1817, in wrap_fx_proxy_cls
    specialized_props = target_cls.specialize(example_value)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/users/anijain/pytorch2/torch/_dynamo/variables/tensor.py", line 175, in specialize
    "has_grad_fn": value.grad_fn is not None,
                   ^^^^^^^^^^^^^
torch._dynamo.exc.InternalTorchDynamoError: Output 0 of TracableCreateParameterBackward is a view and its base or another view of its base has been modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.

from user code:
   File "/data/users/anijain/pytorch2/examples/repro_param_construction.py", line 8, in fn
    return p.add_(1)

Wondering if there is an easy fix?

UPDATE - I tried adding 'clone in the forward of the autograd.Function as the mesasge suggested, but new serious errors popped. So, removing the clone in the latest commit.

[anijain2305]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here