-
Notifications
You must be signed in to change notification settings - Fork 25.6k
[export] turn on hybrid symints by default #130775
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/130775
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New Failures, 1 Unrelated FailureAs of commit 899e3f2 with merge base ef05112 ( NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This pull request was exported from Phabricator. Differential Revision: D59778573 |
@pianpwk has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang Pull Request resolved: #130775 Differential Revision: D59778573 Pulled By: pianpwk
06bbc8e
to
281790a
Compare
This pull request was exported from Phabricator. Differential Revision: D59778573 |
@pianpwk has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
test/export/test_export.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We no longer do any specializations, guards either turn into runtime asserts (if expect_true), or are blocking, and require rewriting or allow_complex_guards_as_runtime_asserts=True
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice. Might be good to name the exported program and then do some test to ensure what runtime assert was added?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I should make a note here, this test case does require the complex guards flag to be set, it encounters a guard Ne(Mod(s0, s0-1), 0)
(special case of reshape) when tracing.
For the cases I've removed, some require this flag, some export fine with just hybrid symints. For those that are fine, the asserts aren't perfect because replacement expressions are set for placeholders, but not checked at runtime. I was planning to leave that as the follow up task (undo replacement, set as runtime assert)
torch/_export/serde/serialize.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is needed for SymBool expressions like Eq(u0, 1)
, where u0 isn't bound to any previous node, and directly calling create_symboolnode()
errors because we haven't tracked range info on u0, so we explicitly deserialize each symbol beforehand.
See test_sym_bool
in test/export/test_serialize.py for example.
torch/export/_trace.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't really necessary, and after #128599 isn't that much more efficient, but if we want to reason more about what guards/runtime asserts to emit, I guess this makes it a bit cleaner to do from the export side. Also I guess there's no point in running this twice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think comments like this are better added to the code itself.
This pull request was exported from Phabricator. Differential Revision: D59778573 |
ec77e87
to
0c08e1e
Compare
torch/export/_trace.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We run the export solver before the runtime asserts pass. Right now this doesn't mean much - the export solver is only there for suggested fixes - we won't even get to constraint solving if that's needed. But if in future we want to control what runtime asserts are emitted for export, or rely on produce_guards()
for some simplification on runtime asserts, I think this makes sense.
Also matches how dynamo currently does it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above, maybe move such comments to code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Storing these flags in ShapeEnvSettings becomes a problem with AOTInductor, where the ShapeEnv is reused from exporting with Hybrid Symints = True, but later is frozen for runtime asserts for Inductor. We can't overwrite the settings, and this fails when we end up deferring another guard we want to keep.
@pianpwk has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
This pull request was exported from Phabricator. Differential Revision: D59778573 |
Summary: Sets `prefer_deferred_runtime_asserts_over_guards=True` for export. Now any guards emitted from `SymNode.expect_true` (e.g. guards implicitly true for ops) won't cause constraint violations, instead appearing in the graph as runtime asserts, or potentially as replacement expressions in the case of equality guards. One example is "complex guards" that must be true for an op to succeed, e.g. ``` x = torch.randn(4, 8) # [s0, s1] y = torch.randn(32) # [s2] out = x.reshape(-1) + y # this emits Eq(s0 * s1, s2), and we represent y's shape as [s0*s1] in the graph, # though we could also keep [s2] and reify the runtime assert. ``` However, doesn't yet handle all complex guards that might cause export to crash - guards emitted from `SymNode.guard_bool/guard_size_oblivious` (e.g. explicit if-else conditions in user code or lower-level op implementations hit during tracing) can still raise constraint violations, and those can be deferred with `allow_complex_guards_as_runtime_asserts=True`. We don't yet turn this on by default because it results in way more asserts in the graph that aren't obvious from the ops at a high-level, and often represent specialization (a variant of the op implementation, or checks for 0/1 specialization) - similar to turning on real tensor propagation. We also remove forced specializations for export - now any guard we can't express with Dims/DerivedDims either are handled with Hybrid SymInts, or should be resolved with rewriting or deferring. Follow up: Currently, `ShapeEnv._set_replacement()` is called for complex equality expressions (e.g. s2 -> s0*s1 in the example above), and the ExportedProgram stores `s0*s1` in the input placeholder. This isn't checked for validity when the program is run, so an option is to avoid/undo replacement and runtime assert on the expression. A similar issue exists for derived dims: if a dim like `3*d0` is a placeholder shape, the value is never checked for divisibility. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang Pull Request resolved: #130775 Differential Revision: D59778573 Pulled By: pianpwk
02a7e62
to
2b63008
Compare
This pull request was exported from Phabricator. Differential Revision: D59778573 |
Summary: Sets `prefer_deferred_runtime_asserts_over_guards=True` for export. Now any guards emitted from `SymNode.expect_true` (e.g. guards implicitly true for ops) won't cause constraint violations, instead appearing in the graph as runtime asserts, or potentially as replacement expressions in the case of equality guards. One example is "complex guards" that must be true for an op to succeed, e.g. ``` x = torch.randn(4, 8) # [s0, s1] y = torch.randn(32) # [s2] out = x.reshape(-1) + y # this emits Eq(s0 * s1, s2), and we represent y's shape as [s0*s1] in the graph, # though we could also keep [s2] and reify the runtime assert. ``` However, doesn't yet handle all complex guards that might cause export to crash - guards emitted from `SymNode.guard_bool/guard_size_oblivious` (e.g. explicit if-else conditions in user code or lower-level op implementations hit during tracing) can still raise constraint violations, and those can be deferred with `allow_complex_guards_as_runtime_asserts=True`. We don't yet turn this on by default because it results in way more asserts in the graph that aren't obvious from the ops at a high-level, and often represent specialization (a variant of the op implementation, or checks for 0/1 specialization) - similar to turning on real tensor propagation. We also remove forced specializations for export - now any guard we can't express with Dims/DerivedDims either are handled with Hybrid SymInts, or should be resolved with rewriting or deferring. Follow up: Currently, `ShapeEnv._set_replacement()` is called for complex equality expressions (e.g. s2 -> s0*s1 in the example above), and the ExportedProgram stores `s0*s1` in the input placeholder. This isn't checked for validity when the program is run, so an option is to avoid/undo replacement and runtime assert on the expression. A similar issue exists for derived dims: if a dim like `3*d0` is a placeholder shape, the value is never checked for divisibility. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang Pull Request resolved: #130775 Reviewed By: avikchaudhuri Differential Revision: D59778573 Pulled By: pianpwk
86f16ba
to
495de52
Compare
495de52
to
187f2cd
Compare
This pull request was exported from Phabricator. Differential Revision: D59778573 |
@pianpwk has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: Sets `prefer_deferred_runtime_asserts_over_guards=True` for export, so any guards emitted from `SymNode.expect_true` (for example, guards that are implicitly required to be true for an op to succeed) won't lead to constraint violations. Instead these should appear in the graph as runtime asserts, or potentially as replacement expressions for placeholder shapes. For example, this reshape op should emit s0 * s1 = s2, deferred as a runtime assert. ``` x = torch.randn(4, 8) # [s0, s1] y = torch.randn(32) # [s2] out = x.reshape(-1) + y # this emits Eq(s0 * s1, s2), and we represent y's shape as [s0*s1] in the graph. ``` However, other complex guards can still cause export to fail, for instance guards emitted from `SymNode.guard_bool/guard_size_oblivious` (e.g. explicit if-else conditions in user code or lower-level op implementations hit during tracing) can still raise constraint violations. These can be deferred with `allow_complex_guards_as_runtime_asserts=True`. We don't yet make this default, because while this makes export more likely to succeed, it results in non-trivial asserts being emitted that often represent specialization to a variant of the op, or checks related to 0/1 specialization. We also remove forced specializations for export and kill the `_disable_forced_specializations` flag - now any guard we can't express with Dims/DerivedDims either are handled with Hybrid SymInts, or should be resolved with rewriting or deferring. Follow up: Currently, `ShapeEnv._set_replacement()` is called for complex equality expressions (e.g. s2 -> s0*s1 in the example above), and the ExportedProgram stores `s0*s1` in the input placeholder. This isn't checked for validity when the program is run, so an option is to avoid replacement and/or runtime assert on equality. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang Pull Request resolved: #130775 Reviewed By: avikchaudhuri Differential Revision: D59778573 Pulled By: pianpwk
This pull request was exported from Phabricator. Differential Revision: D59778573 |
f5f6f67
to
899e3f2
Compare
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 mandatory check(s) failed. The first few are: Dig deeper by viewing the failures on hud |
@pytorchbot merge -f "merged internally, job failures seem unrelated, bc breakage is intentional" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Sets `prefer_deferred_runtime_asserts_over_guards=True` for export, so any guards emitted from `SymNode.expect_true` (for example, guards that are implicitly required to be true for an op to succeed) won't lead to constraint violations. Instead these should appear in the graph as runtime asserts, or potentially as replacement expressions for placeholder shapes. For example, this reshape op should emit s0 * s1 = s2, deferred as a runtime assert. ``` x = torch.randn(4, 8) # [s0, s1] y = torch.randn(32) # [s2] out = x.reshape(-1) + y # this emits Eq(s0 * s1, s2), and we represent y's shape as [s0*s1] in the graph. ``` However, other complex guards can still cause export to fail, for instance guards emitted from `SymNode.guard_bool/guard_size_oblivious` (e.g. explicit if-else conditions in user code or lower-level op implementations hit during tracing) can still raise constraint violations. These can be deferred with `allow_complex_guards_as_runtime_asserts=True`. We don't yet make this default, because while this makes export more likely to succeed, it results in non-trivial asserts being emitted that often represent specialization to a variant of the op, or checks related to 0/1 specialization. We also remove forced specializations for export and kill the `_disable_forced_specializations` flag - now any guard we can't express with Dims/DerivedDims either are handled with Hybrid SymInts, or should be resolved with rewriting or deferring. Follow up: Currently, `ShapeEnv._set_replacement()` is called for complex equality expressions (e.g. s2 -> s0*s1 in the example above), and the ExportedProgram stores `s0*s1` in the input placeholder. This isn't checked for validity when the program is run, so an option is to avoid replacement and/or runtime assert on equality. Pull Request resolved: pytorch#130775 Approved by: https://github.com/avikchaudhuri
Sets `prefer_deferred_runtime_asserts_over_guards=True` for export, so any guards emitted from `SymNode.expect_true` (for example, guards that are implicitly required to be true for an op to succeed) won't lead to constraint violations. Instead these should appear in the graph as runtime asserts, or potentially as replacement expressions for placeholder shapes. For example, this reshape op should emit s0 * s1 = s2, deferred as a runtime assert. ``` x = torch.randn(4, 8) # [s0, s1] y = torch.randn(32) # [s2] out = x.reshape(-1) + y # this emits Eq(s0 * s1, s2), and we represent y's shape as [s0*s1] in the graph. ``` However, other complex guards can still cause export to fail, for instance guards emitted from `SymNode.guard_bool/guard_size_oblivious` (e.g. explicit if-else conditions in user code or lower-level op implementations hit during tracing) can still raise constraint violations. These can be deferred with `allow_complex_guards_as_runtime_asserts=True`. We don't yet make this default, because while this makes export more likely to succeed, it results in non-trivial asserts being emitted that often represent specialization to a variant of the op, or checks related to 0/1 specialization. We also remove forced specializations for export and kill the `_disable_forced_specializations` flag - now any guard we can't express with Dims/DerivedDims either are handled with Hybrid SymInts, or should be resolved with rewriting or deferring. Follow up: Currently, `ShapeEnv._set_replacement()` is called for complex equality expressions (e.g. s2 -> s0*s1 in the example above), and the ExportedProgram stores `s0*s1` in the input placeholder. This isn't checked for validity when the program is run, so an option is to avoid replacement and/or runtime assert on equality. Pull Request resolved: pytorch#130775 Approved by: https://github.com/avikchaudhuri
Summary: #130775 recently stopped killed forced specializations for export on complex guards, so the only way we now get a specialized value is if we're able to solve for it. For example, if we have guards `s0 * 2 = s1`, `s0 + 6 = s1`, we specialize `s0 = 6; s1 = 12`. That might look like this: ``` class Foo(torch.nn.Module): def forward(self, x, y): return x.reshape([-1]) + y dy = Dim("dy", min=6) x, y = torch.randn(6, 2), torch.randn(12) dynamic_shapes = { "x": (dy - 6, 2), "y": (dy,), } ``` Our current error message is: `{symbol} must be specialized to {value} because the guards generated for it are too complex` This is now misleading, so we change it to: `solving the guards generated for {symbol} resulted in a specialized value of {value}` Test Plan: test_export Reviewed By: angelayi Differential Revision: D60787430
#130775 recently killed forced specializations for export on complex guards, so the only way we now get a specialized value is if we're able to solve for it. For example, if we have guards `s0 * 2 = s1`, `s0 + 6 = s1`, we specialize `s0 = 6; s1 = 12`. That might look like this: ``` class Foo(torch.nn.Module): def forward(self, x, y): return x.reshape([-1]) + y dy = Dim("dy", min=6) x, y = torch.randn(6, 2), torch.randn(12) dynamic_shapes = { "x": (dy - 6, 2), "y": (dy,), } ``` Our current error message is: `{symbol} must be specialized to {value} because the guards generated for it are too complex` This is now misleading, so we change it to: `solving the guards generated for {symbol} resulted in a specialized value of {value}` Pull Request resolved: #132698 Approved by: https://github.com/avikchaudhuri
Sets
prefer_deferred_runtime_asserts_over_guards=True
for export, so any guards emitted fromSymNode.expect_true
(for example, guards that are implicitly required to be true for an op to succeed) won't lead to constraint violations. Instead these should appear in the graph as runtime asserts, or potentially as replacement expressions for placeholder shapes.For example, this reshape op should emit s0 * s1 = s2, deferred as a runtime assert.
However, other complex guards can still cause export to fail, for instance guards emitted from
SymNode.guard_bool/guard_size_oblivious
(e.g. explicit if-else conditions in user code or lower-level op implementations hit during tracing) can still raise constraint violations. These can be deferred withallow_complex_guards_as_runtime_asserts=True
. We don't yet make this default, because while this makes export more likely to succeed, it results in non-trivial asserts being emitted that often represent specialization to a variant of the op, or checks related to 0/1 specialization.We also remove forced specializations for export and kill the
_disable_forced_specializations
flag - now any guard we can't express with Dims/DerivedDims either are handled with Hybrid SymInts, or should be resolved with rewriting or deferring.Follow up:
Currently,
ShapeEnv._set_replacement()
is called for complex equality expressions (e.g. s2 -> s0*s1 in the example above), and the ExportedProgram storess0*s1
in the input placeholder. This isn't checked for validity when the program is run, so an option is to avoid replacement and/or runtime assert on equality.cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang