[aot cache][ca] remove restriction on caching ca's aot inference graph #148491
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/148491
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 4175ba9 with merge base 097b0d3 ( UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| ) # from compiled autograd | ||
| with self.assertRaisesRegex( | ||
| torch._dynamo.exc.BackendCompilerFailed, | ||
| "BypassAOTAutogradCache: Unsupported call_function target torch._dynamo.compiled_autograd.ops.validate_outputs", |
There was a problem hiding this comment.
basic q: I would have imagined that the AOTAutograd cache misses, and is forced to re-run AOTAutograd on the ca graph. But in this test it sounds like compile hard errors duration compilation. Is that right / why is that the case?
There was a problem hiding this comment.
yep, we rerun AOTAutograd on the CA dynamo graph, it doesn't fail. Then we try to cache it, which fails because of some dynamically registered ops found in the CA dynamo graph. In this unit test, we set a config to make that a hard error
|
@pytorchbot merge -i |
Merge startedYour change will be merged while ignoring the following 0 checks: Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
|
@pytorchbot rebase |
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
|
Successfully rebased |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: This PR has internal changes and must be landed via Phabricator! Please try reimporting/rexporting the PR! Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
4 participants
Stack from ghstack (oldest at bottom):
but still can't cache CA's aot inference graph yet: the CA functional ops aren't serializable
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames