Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[inductor] Don't inherit __future__ flags from the calling scope when compile -ing generated modules #126454

Closed
wants to merge 5 commits into from
Closed

[inductor] Don't inherit __future__ flags from the calling scope when compile -ing generated modules #126454

wants to merge 5 commits into from

Conversation

amjames
Copy link
Collaborator

@amjames amjames commented May 16, 2024 •
edited

Stack from ghstack (oldest at bottom):

This file includes from __futures__ import annotations which interacts with compile by causing type annotations to be populated as strings. Triton does not parse the string annotation correctly. Avoid this behavior by passing dont_inherit=True to compile.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @desertfire @chauhang

@amjames
Update
03f494f 
[ghstack-poisoned]
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented May 16, 2024 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126454

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 13a5aef with merge base 4b23c4f (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced May 16, 2024
Closed
Closed
@amjames
Update
fc45b0c 
[ghstack-poisoned]
peterbell10
peterbell10 approved these changes May 16, 2024
View reviewed changes
Copy link
Collaborator

@peterbell10 peterbell10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, modulo the lint failures

@amjames
Update
4aebad1 
[ghstack-poisoned]
@amjames
Copy link
Collaborator Author

amjames commented May 17, 2024

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 17, 2024
@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team Raised by workflow job

@peterbell10
Copy link
Collaborator

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

ZelboK pushed a commit to ZelboK/pytorch that referenced this pull request May 19, 2024
@amjames @ZelboK
[inductor] Load python modules using importlib (pytorch#126454)
6e4ed6c 
The `compile` + `exec` workflow is susceptible to behavior drifting from
a "normal" import use importlib instead to avoid this.

In particular here annotations were being stored as strings due to
`from __futures__ import annotations` in the scope calling `compile`.
Triton cares about annotations on global variables and this makes it
much easier to reliably code-gen them.

Pull Request resolved: pytorch#126454
Approved by: https://github.com/peterbell10
@DanilBaibak
Copy link
Contributor

@pytorchbot revert -m "Break internal build" -c ghfirst

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot added a commit that referenced this pull request May 20, 2024
@pytorchmergebot
Revert "[inductor] Load python modules using importlib (#126454)"
5ad2f10 
This reverts commit faa26df.

Reverted #126454 on behalf of https://github.com/DanilBaibak due to Break internal build ([comment](#126454 (comment)))
@pytorchmergebot
Copy link
Collaborator

@amjames your PR has been successfully reverted.

@DanilBaibak
Copy link
Contributor

Hi @amjames! Sorry, I need to revert your PR because it breaks the internal build. Here are the details:

4 TESTS FAILED
  ✗ caffe2/test/inductor:multi_kernel - test_reduction_scratch_buffer_cpp_wrapper (caffe2.test.inductor.test_multi_kernel.MultiKernelTest)
  ✗ caffe2/test/inductor:multi_kernel - test_reduction_scratch_buffer_cpp_wrapper_non_persistent_reduction (caffe2.test.inductor.test_multi_kernel.MultiKernelTest)
  ✗ caffe2/test/inductor:multi_kernel - test_reduction_scratch_buffer_cpp_wrapper_persistent_reduction (caffe2.test.inductor.test_multi_kernel.MultiKernelTest)
  ✗ caffe2/test/inductor:multi_kernel - test_softmax_cpp_wrapper (caffe2.test.inductor.test_multi_kernel.MultiKernelTest)

Traceback:

[2024-05-19T08:42:33.790-07:00]   File "/dev/shm/uid-30083/e06234a2-seed-nspid4026534425_cgpid3536527-ns-4026534755/torch/_inductor/graph.py", line 1738, in compile_to_fn
[2024-05-19T08:42:33.790-07:00]     return self.compile_to_module().call
[2024-05-19T08:42:33.790-07:00]   File "/dev/shm/uid-30083/e06234a2-seed-nspid4026534425_cgpid3536527-ns-4026534755/torch/_inductor/utils.py", line 1018, in patched_compile_to_module
[2024-05-19T08:42:33.790-07:00]     mod = compile_to_module(self)
[2024-05-19T08:42:33.790-07:00]   File "/dev/shm/uid-30083/e06234a2-seed-nspid4026534425_cgpid3536527-ns-4026534755/torch/_dynamo/utils.py", line 210, in time_wrapper
[2024-05-19T08:42:33.790-07:00]     r = func(*args, **kwargs)
[2024-05-19T08:42:33.790-07:00]   File "/dev/shm/uid-30083/e06234a2-seed-nspid4026534425_cgpid3536527-ns-4026534755/torch/_inductor/graph.py", line 1682, in compile_to_module
[2024-05-19T08:42:33.790-07:00]     mod = PyCodeCache.load_by_key_path(
[2024-05-19T08:42:33.790-07:00]   File "/dev/shm/uid-30083/e06234a2-seed-nspid4026534425_cgpid3536527-ns-4026534755/torch/_inductor/codecache.py", line 2584, in load_by_key_path
[2024-05-19T08:42:33.790-07:00]     mod = _reload_python_module(key, path)
[2024-05-19T08:42:33.790-07:00]   File "/dev/shm/uid-30083/e06234a2-seed-nspid4026534425_cgpid3536527-ns-4026534755/torch/_inductor/runtime/compile_tasks.py", line 42, in _reload_python_module
[2024-05-19T08:42:33.790-07:00]     raise RuntimeError(
[2024-05-19T08:42:33.790-07:00] torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
[2024-05-19T08:42:33.790-07:00] RuntimeError: Failed to import /re_tmp/tmppyq1p33x/ca/ccaifp7q25wp7vqbispli5ky6lo5b2qq3zoa75eped3jjeyvrncj.py
[2024-05-19T08:42:33.790-07:00] SkipFrame: COMPILE FAILED WITH EXIT CODE 1
[2024-05-19T08:42:33.790-07:00] stdout: 
[2024-05-19T08:42:33.790-07:00] stderr: 
[2024-05-19T08:42:33.790-07:00]
[2024-05-19T08:42:33.790-07:00] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
[2024-05-19T08:42:33.790-07:00]
[2024-05-19T08:42:33.790-07:00]
[2024-05-19T08:42:33.790-07:00] You can suppress this exception and fall back to eager by setting:
[2024-05-19T08:42:33.790-07:00]     import torch._dynamo
[2024-05-19T08:42:33.790-07:00]     torch._dynamo.config.suppress_errors = True

@amjames
Update
adbff01 
[ghstack-poisoned]
@amjames
Copy link
Collaborator Author

amjames commented May 21, 2024

I reverted back to using compile + exec here, the original issue can be resolved using the dont_inherit kwarg to compile so that is not so bad.

@DanilBaibak do you want to see if those tests fail on the internal build, or should I just try to re-land this after CI is done?

@amjames
Update
13a5aef 
[ghstack-poisoned]
@amjames amjames changed the title [inductor] Load python modules using importlib [inductor] Don't inherit __future__ flags from the calling scope when compile -ing generated modules May 21, 2024
@amjames amjames requested a review from peterbell10 May 21, 2024 13:15
@DanilBaibak
Copy link
Contributor

@amjames, I think you can just re-land after CI is done.

@amjames
Copy link
Collaborator Author

amjames commented May 21, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@kit1980 kit1980 removed the Reverted label May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peterbell10 peterbell10 peterbell10 approved these changes

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor open source topic: not user facing topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@amjames @pytorchmergebot @peterbell10 @DanilBaibak @kit1980 @pytorchbot