Re-land precompile triton templates #124030
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit e0c9764. [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124030
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit f1e2cf9 with merge base bbb6e36 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This was referenced Apr 3, 2024
Closed
Closed
This reverts commit e0c9764. [ghstack-poisoned]
shunting314
approved these changes
Apr 15, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. [ghstack-poisoned]
nmacchioni
approved these changes
Apr 15, 2024
yoyoyocmu
approved these changes
Apr 15, 2024
eellison
added
topic: not user facing
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
pytorchmergebot
pushed a commit
that referenced
this pull request
Apr 17, 2024
Two changes: - in epilogue benchmark fusion, only take top 6 choices. There were basically no choices taken after this in HF. - Share a single precompilation function among matmuls with same key. Pull Request resolved: #122642 Approved by: https://github.com/shunting314 ghstack dependencies: #124030
|
@pytorchbot revert -m "Broken trunk" -c ignoredsignal |
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: pytorch#124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
…2642) Two changes: - in epilogue benchmark fusion, only take top 6 choices. There were basically no choices taken after this in HF. - Share a single precompilation function among matmuls with same key. Pull Request resolved: pytorch#122642 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
…torch#122642)" This reverts commit 050051f. Reverted pytorch#122642 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](pytorch#124030 (comment)))
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
This reverts commit d68196e. Reverted pytorch#124030 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](pytorch#124030 (comment)))
pytorch-bot bot
pushed a commit
that referenced
this pull request
Apr 22, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: #124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
This reverts commit 030bb13. Reverted pytorch#124030 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](pytorch#124030 (comment)))
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: pytorch#124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
…2642) Two changes: - in epilogue benchmark fusion, only take top 6 choices. There were basically no choices taken after this in HF. - Share a single precompilation function among matmuls with same key. Pull Request resolved: pytorch#122642 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
Two small fixes: - preserve rng around compile_fx_inner - Now that will precompile in the background while lowering multiple templates in parallel, we no longer can allocate inputs at the beginning of the function because we will have multiple sets of inputs allocated at the same time. Instead, allocate them when needed. Pull Request resolved: pytorch#123229 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642
sanketpurandare
pushed a commit
to sanketpurandare/pytorch
that referenced
this pull request
Apr 22, 2024
…22825) Two changes: - Make the flag for multi template buffer independent from benchmark fusion. While benchmark fusion can be useful, the compilation time/performance trade offs are different than for just templates, which we'd like to enable by default. - Dont do MultiTemplateBuffers/benchmark-fusion for templates which have custom input gen fn's (currently which only exist internally). Threading the custom input gn fns to benchmark fusion is NYI. Pull Request resolved: pytorch#122825 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642, pytorch#123229
pytorch-bot bot
pushed a commit
that referenced
this pull request
Apr 22, 2024
Biggest movement is 4% HF inference, 9% TIMM inference. Note, this is max-autotune mode so we are more tolerant of compilation increases. We could improve compilation time by limiting: ``` # Take how many of the top triton kernels to benchmark epilogue max_epilogue_benchmarked_choices = 3 ``` There is a hf_Whisper failure which you can repro on main without this stack with `TORCHINDUCTOR_MAX_AUTOTUNE_GEMM_BACKENDS=TRITON TORCHINDUCTOR_MAX_AUTOTUNE=1 python benchmarks/dynamo/torchbench.py --backend inductor --amp --accuracy --training --only hf_Whisper`. When you turn off epilogue fusion, it fixes the accuracy. I bisected the failure to an epilogue, however when you compare the results of that epilogue with the corresponding separate kernels the results of the output are equivalent. Inference: <img width="1686" alt="image" src="https://github.com/pytorch/pytorch/assets/11477974/0b240080-cd33-4c08-89d3-583103b1fb0c"> Training: <img width="1329" alt="Screenshot 2024-04-16 at 6 16 30 PM" src="https://github.com/pytorch/pytorch/assets/11477974/db0afcc9-7288-4c27-84ce-4fc1a5690788"> Pull Request resolved: #124031 Approved by: https://github.com/Chillee, https://github.com/shunting314 ghstack dependencies: #124030, #122642, #123229, #122825
andoorve
pushed a commit
to andoorve/pytorch
that referenced
this pull request
May 1, 2024
This reverts commit 030bb13. Reverted pytorch#124030 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](pytorch#124030 (comment)))
andoorve
pushed a commit
to andoorve/pytorch
that referenced
this pull request
May 1, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: pytorch#124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
andoorve
pushed a commit
to andoorve/pytorch
that referenced
this pull request
May 1, 2024
…2642) Two changes: - in epilogue benchmark fusion, only take top 6 choices. There were basically no choices taken after this in HF. - Share a single precompilation function among matmuls with same key. Pull Request resolved: pytorch#122642 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030
andoorve
pushed a commit
to andoorve/pytorch
that referenced
this pull request
May 1, 2024
Two small fixes: - preserve rng around compile_fx_inner - Now that will precompile in the background while lowering multiple templates in parallel, we no longer can allocate inputs at the beginning of the function because we will have multiple sets of inputs allocated at the same time. Instead, allocate them when needed. Pull Request resolved: pytorch#123229 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642
andoorve
pushed a commit
to andoorve/pytorch
that referenced
this pull request
May 1, 2024
…22825) Two changes: - Make the flag for multi template buffer independent from benchmark fusion. While benchmark fusion can be useful, the compilation time/performance trade offs are different than for just templates, which we'd like to enable by default. - Dont do MultiTemplateBuffers/benchmark-fusion for templates which have custom input gen fn's (currently which only exist internally). Threading the custom input gn fns to benchmark fusion is NYI. Pull Request resolved: pytorch#122825 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642, pytorch#123229
andoorve
pushed a commit
to andoorve/pytorch
that referenced
this pull request
May 1, 2024
Biggest movement is 4% HF inference, 9% TIMM inference. Note, this is max-autotune mode so we are more tolerant of compilation increases. We could improve compilation time by limiting: ``` # Take how many of the top triton kernels to benchmark epilogue max_epilogue_benchmarked_choices = 3 ``` There is a hf_Whisper failure which you can repro on main without this stack with `TORCHINDUCTOR_MAX_AUTOTUNE_GEMM_BACKENDS=TRITON TORCHINDUCTOR_MAX_AUTOTUNE=1 python benchmarks/dynamo/torchbench.py --backend inductor --amp --accuracy --training --only hf_Whisper`. When you turn off epilogue fusion, it fixes the accuracy. I bisected the failure to an epilogue, however when you compare the results of that epilogue with the corresponding separate kernels the results of the output are equivalent. Inference: <img width="1686" alt="image" src="https://github.com/pytorch/pytorch/assets/11477974/0b240080-cd33-4c08-89d3-583103b1fb0c"> Training: <img width="1329" alt="Screenshot 2024-04-16 at 6 16 30 PM" src="https://github.com/pytorch/pytorch/assets/11477974/db0afcc9-7288-4c27-84ce-4fc1a5690788"> Pull Request resolved: pytorch#124031 Approved by: https://github.com/Chillee, https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642, pytorch#123229, pytorch#122825
pytorch-bot bot
pushed a commit
that referenced
this pull request
May 3, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: #124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
…2642) Two changes: - in epilogue benchmark fusion, only take top 6 choices. There were basically no choices taken after this in HF. - Share a single precompilation function among matmuls with same key. Pull Request resolved: pytorch#122642 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
…torch#122642)" This reverts commit 050051f. Reverted pytorch#122642 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](pytorch#124030 (comment)))
pytorch-bot bot
pushed a commit
that referenced
this pull request
May 3, 2024
This reverts commit d68196e. Reverted #124030 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](#124030 (comment)))
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: pytorch#124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
This reverts commit 030bb13. Reverted pytorch#124030 on behalf of https://github.com/DanilBaibak due to Broken trunk ([comment](pytorch#124030 (comment)))
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit. Pull Request resolved: pytorch#124030 Approved by: https://github.com/shunting314, https://github.com/nmacchioni, https://github.com/yoyoyocmu
pytorch-bot bot
pushed a commit
that referenced
this pull request
May 3, 2024
Two changes: - in epilogue benchmark fusion, only take top 6 choices. There were basically no choices taken after this in HF. - Share a single precompilation function among matmuls with same key. Pull Request resolved: #122642 Approved by: https://github.com/shunting314 ghstack dependencies: #124030
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
Two small fixes: - preserve rng around compile_fx_inner - Now that will precompile in the background while lowering multiple templates in parallel, we no longer can allocate inputs at the beginning of the function because we will have multiple sets of inputs allocated at the same time. Instead, allocate them when needed. Pull Request resolved: pytorch#123229 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
…22825) Two changes: - Make the flag for multi template buffer independent from benchmark fusion. While benchmark fusion can be useful, the compilation time/performance trade offs are different than for just templates, which we'd like to enable by default. - Dont do MultiTemplateBuffers/benchmark-fusion for templates which have custom input gen fn's (currently which only exist internally). Threading the custom input gn fns to benchmark fusion is NYI. Pull Request resolved: pytorch#122825 Approved by: https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642, pytorch#123229
petrex
pushed a commit
to petrex/pytorch
that referenced
this pull request
May 3, 2024
Biggest movement is 4% HF inference, 9% TIMM inference. Note, this is max-autotune mode so we are more tolerant of compilation increases. We could improve compilation time by limiting: ``` # Take how many of the top triton kernels to benchmark epilogue max_epilogue_benchmarked_choices = 3 ``` There is a hf_Whisper failure which you can repro on main without this stack with `TORCHINDUCTOR_MAX_AUTOTUNE_GEMM_BACKENDS=TRITON TORCHINDUCTOR_MAX_AUTOTUNE=1 python benchmarks/dynamo/torchbench.py --backend inductor --amp --accuracy --training --only hf_Whisper`. When you turn off epilogue fusion, it fixes the accuracy. I bisected the failure to an epilogue, however when you compare the results of that epilogue with the corresponding separate kernels the results of the output are equivalent. Inference: <img width="1686" alt="image" src="https://github.com/pytorch/pytorch/assets/11477974/0b240080-cd33-4c08-89d3-583103b1fb0c"> Training: <img width="1329" alt="Screenshot 2024-04-16 at 6 16 30 PM" src="https://github.com/pytorch/pytorch/assets/11477974/db0afcc9-7288-4c27-84ce-4fc1a5690788"> Pull Request resolved: pytorch#124031 Approved by: https://github.com/Chillee, https://github.com/shunting314 ghstack dependencies: pytorch#124030, pytorch#122642, pytorch#123229, pytorch#122825
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Re-land precompile triton templates. This got reverted because we were precompiling templates without checking the cache. I have since added logic and a test to ensure we do not precompile if there is a cache hit.
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @amjames @desertfire @chauhang