[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxruntime.py #99434

titaiwangms · 2023-04-18T18:02:08Z

Stack from ghstack (oldest at bottom):

-> [ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxruntime.py #99434

Expand additional_test_inputs to include kwargs
Revisit and update tests status by adding ops
Disabling dtype -1 assignment avoids potential bugs
Expand input/output to accept buit-in type, but they are not dynamically captured by dynamo.export right now, and they would be added as constant input to op.targets.
Move run_test_with_fx_to_onnx_exporter_and_onnx_runtime to onnx_test_common.py

`🤖 Generated by Copilot at 3c03579`

Summary

🛠️🧪🚀

This pull request improves the ONNX export support for scalar types and some ATen operators in PyTorch. It updates the test framework, the input and output adapters, the function dispatcher and the ONNX script generator to handle these cases. It also fixes or removes some failing or outdated tests.

We defy the limits of the ONNX script
We export the models with scalar and copy
We filter and convert the kwargs of dtype
We run the tests with FX and docstring

Walkthrough

Update the _InputArgsType type annotation and the _run_test_with_fx_to_onnx_exporter_and_onnx_runtime function signature and docstring to handle int, float and bool inputs for some ONNX operators (link, link, link, link, link, link, link)
Update the filter_incompatible_and_dtype_convert_kwargs function to omit the dtype argument if it is None (link)
Update the test cases in test_fx_to_onnx_with_onnxruntime.py to use the input_kwargs parameter as a mapping, to fix the format of the additional_test_inputs parameter, and to add or remove xfail, skip_dynamic_fx_test and skip_min_ort_version decorators as needed (link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link, link)

…me.py [ghstack-poisoned]

pytorch-bot · 2023-04-18T18:02:11Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99434

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

ROCm jobs are failing with No space left on device

✅ No Failures

As of commit f836124:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…me.py ghstack-source-id: d7808fa51412e55efd4234f10adce0ee30ab23dd Pull Request resolved: #99434

titaiwangms · 2023-04-18T18:03:46Z

copilot:all

…h_onnxruntime.py" [ghstack-poisoned]

…me.py ghstack-source-id: 01716dab3990b5845ff82d2a4e6bd25d88084436 Pull Request resolved: #99434

test/onnx/test_fx_to_onnx_with_onnxruntime.py

BowenBao · 2023-04-18T21:29:59Z

copilot:all

Has to be in the main pr description section

…me.py ghstack-source-id: 9859f4624ce929f446b6783b96fbc4e844ee08cd Pull Request resolved: #99434

Should be merged after #99434 [ghstack-poisoned]

…me.py ghstack-source-id: 0746b45f7f44a64b8bd5769347fa76691c654e80 Pull Request resolved: #99434

torch/onnx/_internal/exporter.py

test/onnx/test_fx_to_onnx_with_onnxruntime.py

justinchuby · 2023-04-19T21:47:48Z

test/onnx/test_fx_to_onnx_with_onnxruntime.py

        )

    @pytorch_test_common.xfail(
-        "RuntimeError: Unknown call_function target: aten.copy.default"
+        "RuntimeError: Unknown call_function target: aten.lift_fresh_copy.default"


👀 is this a new op in the family

Looks like it's just not implemented yet.

justinchuby · 2023-04-19T21:51:00Z

test/onnx/onnx_test_common.py

@@ -140,3 +154,199 @@ def _run_test(m, remained_onnx_input_idx, flatten=True, ignore_none=True):
            )
        if not is_model_script and not self.is_script:
            _run_test(model, tracing_remained_onnx_input_idx)
+
+    @_beartype.beartype
+    def run_test_with_fx_to_onnx_exporter_and_onnx_runtime(


Does this need to be a method in the test class or can it be a global function? Prefer a global function if it does not require states in self?

hmmm thought about that, but the function requires test_suite attributes: op_level_debug and dynamic_shapes. We could move it globally, but will need to impot class TestFx... to annotate, which is cons.

SG! This is good then imo

titaiwangms · 2023-04-19T23:06:12Z

@BowenBao PTAL

…me.py ghstack-source-id: d3245ec44db2d0d9a464943e1f4b557a336e84cd Pull Request resolved: #99434

titaiwangms · 2023-04-21T20:44:29Z

test/onnx/test_fx_to_onnx_with_onnxruntime.py

-        "typing.Sequence[torch.Tensor], as tuple index 1 item float 8.0 not "
-        "instance of <protocol 'torch.Tensor'>."
+        "https://github.com/pytorch/pytorch/issues/99534"
+        "To make it work, convert the float argument into a 0d tensor"


I wonder if we still want to keep this test. xfail seems fine.

titaiwangms · 2023-04-22T01:44:03Z

@pytorchbot merge

pytorchmergebot · 2023-04-22T01:47:19Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Should be merged after #99434  ### <samp>🤖 Generated by Copilot at f59c428</samp> ### Summary 📝🔄🚀  This pull request enhances the ONNX exporter and its tests to handle more input and output types for PyTorch models. It improves the type annotation, tolerance handling, and documentation of the exporter and its test functions. It also renames a test function to make the code more consistent and clear. > _To export PyTorch models with ONNX_ > _We need to handle various contexts_ > _We improved the annotation_ > _And the tolerance function_ > _And renamed `dont_care` to `skip` for the tests_ ### Walkthrough * Expand the `_InputArgsType` type annotation to include int, float, and bool types, in addition to torch.Tensor, Sequence, and Mapping, to support more types of inputs for PyTorch models ([link](https://github.com/pytorch/pytorch/pull/99465/files?diff=unified&w=0#diff-c8fa56eefd7f98fb4f9739d57df57f02ede77e28528133736010a6d06651ebcbL44-R46), [link](https://github.com/pytorch/pytorch/pull/99465/files?diff=unified&w=0#diff-0795f54fd1f38cfbf2c4a863a4efc9f40f2ea020a2b1612605c361b8d8d35862L197-R199), [link](https://github.com/pytorch/pytorch/pull/99465/files?diff=unified&w=0#diff-0795f54fd1f38cfbf2c4a863a4efc9f40f2ea020a2b1612605c361b8d8d35862L291-R295)). [ghstack-poisoned]

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxrunti…

63ce5a2

…me.py [ghstack-poisoned]

titaiwangms requested review from BowenBao and abock as code owners April 18, 2023 18:02

pytorch-bot bot added the topic: not user facing topic category label Apr 18, 2023

titaiwangms added a commit that referenced this pull request Apr 18, 2023

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxrunti…

e566bac

…me.py ghstack-source-id: d7808fa51412e55efd4234f10adce0ee30ab23dd Pull Request resolved: #99434

titaiwangms added module: onnx Related to torch.onnx release notes: onnx torch.onnx related changes that should show up in the release notes labels Apr 18, 2023

Update on "[ONNX] Add additional_test_kwargs into test_fx_to_onnx_wit…

3c03579

…h_onnxruntime.py" [ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Apr 18, 2023

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxrunti…

81a5cd9

…me.py ghstack-source-id: 01716dab3990b5845ff82d2a4e6bd25d88084436 Pull Request resolved: #99434

pytorchbot added the open source label Apr 18, 2023

BowenBao reviewed Apr 18, 2023

View reviewed changes

test/onnx/test_fx_to_onnx_with_onnxruntime.py Show resolved Hide resolved

titaiwangms added a commit that referenced this pull request Apr 19, 2023

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxrunti…

703824c

…me.py ghstack-source-id: 9859f4624ce929f446b6783b96fbc4e844ee08cd Pull Request resolved: #99434

titaiwangms mentioned this pull request Apr 19, 2023

[ONNX] Add test_fx_op_consistency.py #99465

Closed

titaiwangms added a commit that referenced this pull request Apr 19, 2023

Update base for Update on "[ONNX] Add test_fx_op_consistency.py"

7523cac

Should be merged after #99434 [ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Apr 19, 2023

Update on "[ONNX] Add test_fx_op_consistency.py"

f59c428

Should be merged after #99434 [ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Apr 19, 2023

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxrunti…

3116f95

…me.py ghstack-source-id: 0746b45f7f44a64b8bd5769347fa76691c654e80 Pull Request resolved: #99434

titaiwangms requested a review from justinchuby April 19, 2023 21:16

titaiwangms commented Apr 19, 2023

View reviewed changes

torch/onnx/_internal/exporter.py Outdated Show resolved Hide resolved

justinchuby reviewed Apr 19, 2023

View reviewed changes

test/onnx/test_fx_to_onnx_with_onnxruntime.py Show resolved Hide resolved

justinchuby reviewed Apr 19, 2023

View reviewed changes

justinchuby approved these changes Apr 19, 2023

View reviewed changes

justinchuby reviewed Apr 19, 2023

View reviewed changes

titaiwangms added a commit that referenced this pull request Apr 21, 2023

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxrunti…

c53d8df

…me.py ghstack-source-id: d3245ec44db2d0d9a464943e1f4b557a336e84cd Pull Request resolved: #99434

titaiwangms commented Apr 21, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 22, 2023

pytorchmergebot added the merging label Apr 22, 2023

pytorchmergebot added the Merged label Apr 22, 2023

pytorchmergebot closed this in bd71911 Apr 22, 2023

facebook-github-bot deleted the gh/titaiwangms/6/head branch June 8, 2023 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxruntime.py #99434

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxruntime.py #99434

titaiwangms commented Apr 18, 2023 •

edited

pytorch-bot bot commented Apr 18, 2023 •

edited

titaiwangms commented Apr 18, 2023

BowenBao commented Apr 18, 2023

justinchuby Apr 19, 2023

titaiwangms Apr 21, 2023

justinchuby Apr 19, 2023

titaiwangms Apr 19, 2023

justinchuby Apr 19, 2023

titaiwangms commented Apr 19, 2023

titaiwangms Apr 21, 2023

titaiwangms commented Apr 22, 2023

pytorchmergebot commented Apr 22, 2023

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxruntime.py #99434

[ONNX] Add additional_test_kwargs into test_fx_to_onnx_with_onnxruntime.py #99434

Conversation

titaiwangms commented Apr 18, 2023 • edited

🤖 Generated by Copilot at 3c03579

Summary

Walkthrough

pytorch-bot bot commented Apr 18, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99434

❗ 1 Active SEVs

✅ No Failures

titaiwangms commented Apr 18, 2023

BowenBao commented Apr 18, 2023

justinchuby Apr 19, 2023

Choose a reason for hiding this comment

titaiwangms Apr 21, 2023

Choose a reason for hiding this comment

justinchuby Apr 19, 2023

Choose a reason for hiding this comment

titaiwangms Apr 19, 2023

Choose a reason for hiding this comment

justinchuby Apr 19, 2023

Choose a reason for hiding this comment

titaiwangms commented Apr 19, 2023

titaiwangms Apr 21, 2023

Choose a reason for hiding this comment

titaiwangms commented Apr 22, 2023

pytorchmergebot commented Apr 22, 2023

Merge started

titaiwangms commented Apr 18, 2023 •

edited

`🤖 Generated by Copilot at 3c03579`

pytorch-bot bot commented Apr 18, 2023 •

edited