Various OpInfo architecture improvements #75951

mruberry · 2022-04-18T00:19:16Z

This PR makes the following improvements:

moves the custom skip list for test_normalize_operator_exhaustive in test_fx_experimental to use the typical OpInfo skip architecture. The skips were updated to xfails, and that identified some operators which were no longer failing the test
redundant tests with OpInfo-based testing in test_jit.py were removed
test_dtypes was improved so its error messages are clear and it makes test_nondifferentiable redundant; the latter test has been removed
OpInfo.supports_complex_autograd() is removed in favor of a more accurate and general test for whether the particular dtype is in the backward dtypes of the operator
gradchecks have been improved to verify that an operator doesn't support grad if it claims not to
gradchecks have been improved to test the gradient of all input tensors that require gradient
the concept of "default test dtypes" has been removed
excessive and mostly redundant out testing for elementwise unary operators has been removed
metadata for whether an op supports nuanced "safe casting" to out behavior has been removed from OpInfos
numerous skips have been converted to xfails
numerous OpInfos have had their metadata fixed based on the new checks
jit-specific utilities in common_methods_invocations.py have been moved to jit_programming_utils.py

…_arch

facebook-github-bot · 2022-04-18T00:19:22Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/75951
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
❓Need help or want to give feedback on the CI? Visit our office hours

💊 CI failures summary and remediations

As of commit 06f225f (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-18T18:03:20.4598872Z The PR is introduc...m to confirm whether this change is wanted or not.

2022-04-18T18:03:20.4583713Z processing existing schema:  text(__torch__.torch.classes.profiling.SourceRef _0) -> (str _0)
2022-04-18T18:03:20.4585062Z processing existing schema:  count(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-18T18:03:20.4586781Z processing existing schema:  duration_ns(__torch__.torch.classes.profiling.InstructionStats _0) -> (int _0)
2022-04-18T18:03:20.4588595Z processing existing schema:  source(__torch__.torch.classes.profiling.SourceStats _0) -> (__torch__.torch.classes.profiling.SourceRef _0)
2022-04-18T18:03:20.4591016Z processing existing schema:  line_map(__torch__.torch.classes.profiling.SourceStats _0) -> (Dict(int, __torch__.torch.classes.profiling.InstructionStats) _0)
2022-04-18T18:03:20.4591950Z processing existing schema:  __init__(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-18T18:03:20.4593706Z processing existing schema:  enable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-18T18:03:20.4595012Z processing existing schema:  disable(__torch__.torch.classes.profiling._ScriptProfile _0) -> (NoneType _0)
2022-04-18T18:03:20.4597606Z processing existing schema:  _dump_stats(__torch__.torch.classes.profiling._ScriptProfile _0) -> (__torch__.torch.classes.profiling.SourceStats[] _0)
2022-04-18T18:03:20.4598487Z processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> (NoneType _0)
2022-04-18T18:03:20.4598872Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2022-04-18T18:03:20.4598882Z 
2022-04-18T18:03:20.4598987Z Broken ops: [
2022-04-18T18:03:20.4599467Z 	aten::_validate_sparse_compressed_tensor_args(Tensor crow_indices, Tensor col_indices, Tensor values, int[] size, int layout) -> ()
2022-04-18T18:03:20.4599547Z ]
2022-04-18T18:03:20.5577356Z + cleanup
2022-04-18T18:03:20.5577469Z + retcode=1
2022-04-18T18:03:20.5577575Z + set +x
2022-04-18T18:03:20.5610791Z ##[error]Process completed with exit code 1.
2022-04-18T18:03:20.5650090Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-04-18T18:03:20.5650167Z with:

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

ngimel

Those are cool improvements and simplifications!

ngimel · 2022-04-18T02:02:03Z

test/test_jit.py

 S = 5

-
-def add_nn_functional_test(name, self_size, args, variant_name='', check_ad=(), skipTestIf=(),


is this now done as part of OpInfo testing?

It has been for some time

ngimel · 2022-04-18T02:11:34Z

test/test_ops.py

+        claimed_but_unsupported_backward = claimed_backward & unsupported_backward
+
+        # Partially supporting a dtype is not an error, but we print a warning
+        if (len(partially_supported_forward) + len(partially_supported_backward)) > 0:


or, partially supporting dtype is no longer an error? nice!

Yep! And the error (and warning) messages are now much clearer

ngimel · 2022-04-18T02:13:17Z

test/test_ops.py

+
+        # Partially supporting a dtype is not an error, but we print a warning
+        if (len(partially_supported_forward) + len(partially_supported_backward)) > 0:
+            msg = "Some dtypes for {0} on device type {1} are only partially supported!\n".format(


are supported only for some sample inputs

ngimel · 2022-04-18T02:14:35Z

test/test_ops.py

+                op.name, device_type
+            )
+            if len(partially_supported_forward) > 0:
+                msg = msg + "The following dtypes only worked on some samples during forward: {0}.\n".format(


sample inputs

ngimel · 2022-04-18T02:57:46Z

test/test_ops_gradients.py

+
+            # Creates gradcheck inputs by identifying tensors requiring grad
+            all_args = None
+            if is_iterable_of_tensors(sample.input):


do we still have sample.inputs that are iterables rather than single tensors?

ngimel · 2022-04-18T02:58:40Z

test/test_ops_gradients.py

+            if is_iterable_of_tensors(sample.input):
+                all_args = chain(sample.input, sample.args, sample.kwargs.values())
+            else:
+                all_args = tuple(chain((sample.input,), sample.args, sample.kwargs.values()))


do you need tuple here?

ngimel · 2022-04-18T03:14:54Z

test/test_unary_ufuncs.py

-            self.assertEqual(output, expected.to(output.dtype))
-
-    @ops(unary_ufuncs, dtypes=OpDTypes.supported)
-    def test_out_arg_all_dtypes(self, device, dtype, op):


is this checked in test_out? Not for all dtypes, I'm sure, but that's probably fine

We don't check safe casting to a higher dtype in test_out yet, no

ngimel · 2022-04-18T03:33:27Z

torch/testing/_internal/common_methods_invocations.py

@@ -9560,14 +9530,20 @@ def generate_std_var_kwargs(t: torch.Tensor, **kwargs):
           supports_forward_ad=True,
           supports_fwgrad_bwgrad=True,
           skips=(
+               # Float did not match double
+               DecorateInfo(unittest.expectedFailure, 'TestGradients', 'test_fn_grad'),


so gradients are all wrong here?

As best the test suite can tell: yes

Nice. Is there a hi pri issue about this, or we don't care about cov (or whatever this function is)?

ngimel · 2022-04-18T03:37:22Z

torch/testing/_internal/common_methods_invocations.py

                   supports_forward_ad=True,
                   supports_fwgrad_bwgrad=True,
                   sample_inputs_func=sample_inputs_polygamma,
                   skips=(
                       # Redundant tests
                       DecorateInfo(unittest.skip("Skipped!"), 'TestGradients'),
                       DecorateInfo(unittest.skip("Skipped!"), 'TestJit'),
+                       DecorateInfo(unittest.skip("Skipped!"), 'TestNormalizeOperators'),


Normalize is probably not a redundant test?

ngimel · 2022-04-18T16:33:16Z

torch/testing/_internal/common_methods_invocations.py

-               DecorateInfo(unittest.expectedFailure, 'TestCudaFuserOpInfo', 'test_nvfuser_correctness', dtypes=(torch.bfloat16,)),
+               DecorateInfo(unittest.skip("Works on some configs"), 'TestNNCOpInfo',
+                            'test_nnc_correctness', dtypes=(torch.bfloat16,)),
+               DecorateInfo(unittest.skip("Works on some conifgs"), 'TestCudaFuserOpInfo',


haha, probably on configs that are not supported by nvfuser

mruberry · 2022-04-18T21:54:03Z

@pytorchbot merge this please

github-actions · 2022-04-18T21:56:20Z

Hey @mruberry.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

ngimel · 2022-04-18T23:36:30Z

torch/testing/_internal/common_methods_invocations.py

                                                  torch.complex64: 1e-2}),),
                   skips=(
+                       # Failing with wrong imaginary sign on at least some Windows jobs
+                       DecorateInfo(unittest.skip("Skipped!"), 'TestUnaryUfuncs', 'test_reference_numerics_normal',


this is a year old issue #52299, it's weird that it's still not fixed, are our windows jobs using some old cuda version?

Summary: This PR makes the following improvements: - moves the custom skip list for test_normalize_operator_exhaustive in test_fx_experimental to use the typical OpInfo skip architecture. The skips were updated to xfails, and that identified some operators which were no longer failing the test - redundant tests with OpInfo-based testing in test_jit.py were removed - test_dtypes was improved so its error messages are clear and it makes test_nondifferentiable redundant; the latter test has been removed - OpInfo.supports_complex_autograd() is removed in favor of a more accurate and general test for whether the particular dtype is in the backward dtypes of the operator - gradchecks have been improved to verify that an operator doesn't support grad if it claims not to - gradchecks have been improved to test the gradient of all input tensors that require gradient - the concept of "default test dtypes" has been removed - excessive and mostly redundant out testing for elementwise unary operators has been removed - metadata for whether an op supports nuanced "safe casting" to out behavior has been removed from OpInfos - numerous skips have been converted to xfails - numerous OpInfos have had their metadata fixed based on the new checks - jit-specific utilities in common_methods_invocations.py have been moved to jit_programming_utils.py Pull Request resolved: #75951 Approved by: https://github.com/ngimel Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/de949a0e59f458a77c213d270106a0c9fea6c484 Reviewed By: seemethere Differential Revision: D35751451 fbshipit-source-id: 3bb18586388d7b815a3e4d999769d05082f6d4eb

This PR makes the following improvements: - moves the custom skip list for test_normalize_operator_exhaustive in test_fx_experimental to use the typical OpInfo skip architecture. The skips were updated to xfails, and that identified some operators which were no longer failing the test - redundant tests with OpInfo-based testing in test_jit.py were removed - test_dtypes was improved so its error messages are clear and it makes test_nondifferentiable redundant; the latter test has been removed - OpInfo.supports_complex_autograd() is removed in favor of a more accurate and general test for whether the particular dtype is in the backward dtypes of the operator - gradchecks have been improved to verify that an operator doesn't support grad if it claims not to - gradchecks have been improved to test the gradient of all input tensors that require gradient - the concept of "default test dtypes" has been removed - excessive and mostly redundant out testing for elementwise unary operators has been removed - metadata for whether an op supports nuanced "safe casting" to out behavior has been removed from OpInfos - numerous skips have been converted to xfails - numerous OpInfos have had their metadata fixed based on the new checks - jit-specific utilities in common_methods_invocations.py have been moved to jit_programming_utils.py Pull Request resolved: #75951 Approved by: https://github.com/ngimel (cherry picked from commit de949a0)

This re-enables the test_reference_numerics_extremal tests for atan and atanh. This PR limits the tests to be not run with complex128 as its still failing. These tests were disabled previously in this PR: #75951 There is no open issue for these tests. Signed-off-by: Arindam Roy <rarindam@gmail.com> Pull Request resolved: #77669 Approved by: https://github.com/mruberry

Summary: This re-enables the test_reference_numerics_extremal tests for atan and atanh. This PR limits the tests to be not run with complex128 as its still failing. These tests were disabled previously in this PR: #75951 There is no open issue for these tests. Signed-off-by: Arindam Roy <rarindam@gmail.com> Pull Request resolved: #77669 Approved by: https://github.com/mruberry Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/1af47a3a3e55d2dda094a12a26e603a1c7b5e009 Reviewed By: seemethere, b0noI Differential Revision: D36493970 fbshipit-source-id: 389653eaea304212d1a0a821c3023fcd2e296deb

Mike Ruberry added 3 commits April 15, 2022 18:20

stashes

81dc6d8

Merge branch 'master' of ssh://github.com/pytorch/pytorch into opinfo…

71bf217

…_arch

update

ef1e5b6

facebook-github-bot added the cla signed label Apr 18, 2022

mruberry requested a review from ngimel April 18, 2022 00:19

mruberry added the ciflow/all label Apr 18, 2022

Mike Ruberry added 5 commits April 17, 2022 17:43

include fixes

0b97651

opinfo fixes

fad4a7d

test fixes

552fb44

fixes

795db14

lint

b908da1

ngimel approved these changes Apr 18, 2022

View reviewed changes

Mike Ruberry added 5 commits April 17, 2022 21:58

reduction fixes

fbf871c

fixes

c9787f2

fix

71bd34d

fixes

6833000

fixes

03b8212

ngimel reviewed Apr 18, 2022

View reviewed changes

fixes

06f225f

pytorchmergebot closed this in de949a0 Apr 18, 2022

ngimel reviewed Apr 18, 2022

View reviewed changes

kshitij12345 mentioned this pull request Apr 19, 2022

[testing] deprecate field default_test_dtypes from OpInfo #75886

Closed

mruberry added topic: not user facing topic category module: primTorch labels May 2, 2022

arindamroy-eng mentioned this pull request May 17, 2022

ROCM: Enable few more tests for ROCM #77669

Closed

mruberry deleted the opinfo_arch branch May 19, 2022 19:00

bmedishe mentioned this pull request Nov 2, 2022

Sow ms7 enable test reference numerics small all except complex ROCm/pytorch#1134

Closed

		S = 5


		def add_nn_functional_test(name, self_size, args, variant_name='', check_ad=(), skipTestIf=(),

Various OpInfo architecture improvements #75951

Various OpInfo architecture improvements #75951

Uh oh!

Conversation

mruberry commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

pull / linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Uh oh!

ngimel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mruberry commented Apr 18, 2022

Uh oh!

github-actions bot commented Apr 18, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mruberry commented Apr 18, 2022 •

edited

Loading

facebook-github-bot commented Apr 18, 2022 •

edited

Loading