[dynamo][eval_frame] Create a dynamic wrapper fn to avoid cache collisions

[anijain2305]

Stack from ghstack (oldest at bottom):

• -> [dynamo][eval_frame] Create a dynamic wrapper fn to avoid cache collisions #124881
• [dynamo] Allow inlining of hooks for the top module #124501

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124881

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 1 Unrelated Failure

As of commit e1bf938 with merge base 90258e8 ():

NEW FAILURES - The following jobs have failed:

• inductor / rocm6.0-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2) (gh)
  ##[error]The operation was canceled.
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding
• pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)
  test_resnet18
• pull / linux-focal-py3.11-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding
• pull / linux-focal-py3.12-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding
• pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding
• pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, linux.4xlarge) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge) (gh)
  dynamo/test_hooks.py::HooksTests::test_register_hook_partial_guarding

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
  sebotnet33ts_256

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[anijain2305]

Dynamo kicks in while creating the new function.

[anijain2305]

Closing this PR after iterating it over for a week.

Why this PR?
When we wrap_inline, we want to create a new code object to avoid Dynamo collisions. This makes sense.

Whats the problem?
If we have two different instances of nn.Module. And we put torch.compile on both of them. Today, we will see one compilation (with inlined inbuilt nn modules) because the cache sits on forward.__code__ (or on the inner.code) object. But with this change, there will be a full recompilation. Now, this can be solved by using cache of wrappers. But then it complicates the impl because we need to develop a key for this new cache. Simply caching on code object won't work because __closure__ can change between different functions. We can make more and more comprehensive cache, but given that we call this wrapper infrequently, its fine to just leave it as is.