Make CI less noisy

[tugsbayasgalan]

Stack from ghstack (oldest at bottom):

• -> Make CI less noisy #124664
• Some update #124450
• Experimental prototype for converting torch.jit.trace modules to export #124449

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang

Differential Revision: D56440612

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124664

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 60 New Failures, 2 Unrelated Failures

As of commit defc50a with merge base 00dd4d5 ():

NEW FAILURES - The following jobs have failed:

• inductor / rocm6.0-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2) (gh)
  test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_batch_norm_cuda_float32
• Lint / lintrunner-clang / linux-job (gh)
  >>> Lint for torch/csrc/jit/python/init.cpp:
• Lint / lintrunner-noclang / linux-job (gh)
  >>> Lint for torch/jit/_trace.py:
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_tensorboard.py::TestTensorBoardPytorchGraph::test_torchvision_smoke
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_zeros_like_cuda_float32
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 4, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_zeros_like_cuda_complex64
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_jit_fuser_te.py::TestTEFuserDynamic::test_type_as_cat
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (deploy, 1, 1, linux.4xlarge.nvidia.gpu) (gh)
  RuntimeError: TORCH EXPORT SUCCEEDED
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_zeros_like_cuda_float32
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_quantization.py::TestQuantizeDynamicJitOps::test_embedding_bag_padding_idx_error
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_jit_fuser_te.py::TestTEFuserDynamic::test_where_ops
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  test_ops_jit.py::TestJitCUDA::test_variant_consistency_jit_vsplit_cuda_float32
• pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_zeros_like_cpu_complex64
• pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  test_quantization.py::TestQuantizeDynamicJitOps::test_embedding_bag_padding_idx_error
• pull / linux-focal-py3.11-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-py3.11-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  test_jit_autocast.py::TestJitTraceAutocast::test_nhwc_autocast_jit_trace_model
• pull / linux-focal-py3.11-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  test_jit_llga_fuser.py::TestOpCPU::test_variable_kernel_avg_pool2d_cpu_float32
• pull / linux-focal-py3.11-clang10 / test (dynamo, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-py3.11-clang10 / test (dynamo, 2, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_sample
• pull / linux-focal-py3.11-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh)
  test_jit_fuser_te.py::TestLoopnestRandomizationCPU::test_relu_cpu
• pull / linux-focal-py3.12-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-py3.12-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  test_jit_fuser_te.py::TestLoopnestRandomizationCPU::test_relu_cpu
• pull / linux-focal-py3.12-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_zeros_like_cpu_float32
• pull / linux-focal-py3.12-clang10 / test (dynamo, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-py3.12-clang10 / test (dynamo, 2, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_sample
• pull / linux-focal-py3.12-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh)
  test_jit_fuser_te.py::TestLoopnestRandomizationCPU::test_relu_cpu
• pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_zeros_like_cpu_complex64
• pull / linux-focal-py3.8-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  test_quantization.py::TestQuantizeDynamicJitOps::test_embedding_bag_padding_idx_error
• pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  test_jit_fuser_te.py::TestLoopnestRandomizationCPU::test_relu_cpu
• pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_zeros_like_cpu_float32
• pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-focal-py3.8-clang10 / test (dynamo, 2, 3, linux.2xlarge) (gh)
  test_nn.py::TestNNDeviceTypeCPU::test_upsamplingBiMode2d_nonsupported_dtypes_antialias_False_num_channels_3_mode_bicubic_int16_cpu_int16
• pull / linux-focal-py3.8-clang10 / test (dynamo, 3, 3, linux.2xlarge) (gh)
  test_jit_fuser_te.py::TestLoopnestRandomizationCPU::test_relu_cpu
• pull / linux-focal-py3.8-clang10-onnx / test (default, 1, 2, linux.2xlarge) (gh)
  onnx/test_pytorch_onnx_onnxruntime.py::TestONNXRuntime_opset_version_18_is_script_False_keep_initializers_as_inputs_False::test_qat_linear_relu_fused
• pull / linux-focal-py3.8-clang10-onnx / test (default, 2, 2, linux.2xlarge) (gh)
  onnx/test_pytorch_onnx_onnxruntime.py::TestONNXRuntime_opset_version_18_is_script_False_keep_initializers_as_inputs_True::test_qat_linear_relu_fused
• pull / linux-jammy-py3-clang12-executorch / build (gh)
  Process completed with exit code 1.
• pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, linux.4xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, linux.4xlarge) (gh)
  test_tensorboard.py::TestTensorBoardPytorchGraph::test_torchvision_smoke
• pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, linux.4xlarge) (gh)
  test_jit_fuser_te.py::TestTEFuserDynamic::test_relu
• pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, linux.4xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_view_cpu_float32
• pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, linux.4xlarge) (gh)
  test_jit_fuser_te.py::TestTEFuserDynamic::test_where_ops
• pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, linux.4xlarge) (gh)
  test_jit_llga_fuser.py::TestOpCPU::test_variable_kernel_avg_pool2d_cpu_float32
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• pull / linux-jammy-py3.8-gcc11 / test (default, 2, 3, linux.2xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_softmax_cpu_float32
• pull / linux-jammy-py3.8-gcc11 / test (default, 3, 3, linux.2xlarge) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit_triangular_solve_cpu_complex64
• pull / linux-jammy-py3.8-gcc11 / test (jit_legacy, 1, 1, linux.2xlarge) (gh)
  test_jit_legacy.py::TestScript::test_zero_dimension_tensor_trace
• trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (jit_legacy, 1, 1, linux.4xlarge.nvidia.gpu) (gh)
  test_jit_legacy.py::TestScript::test_zero_dimension_tensor_trace
• trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 1, linux.2xlarge) (gh)
  dynamo/test_interop.py::InteropTests::test_trace_fn
• trunk / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 1, linux.2xlarge) (gh)
  dynamo/test_interop.py::InteropTests::test_trace_fn
• trunk / linux-focal-rocm6.0-py3.8 / test (default, 1, 1, linux.rocm.gpu) (gh)
  test_autograd.py::TestAutograd::test_profiler_propagation
• trunk / macos-12-py3-arm64 / test (default, 1, 3, macos-m1-stable) (gh)
  distributions/test_distributions.py::TestJit::test_variance
• trunk / macos-12-py3-arm64 / test (default, 2, 3, macos-m1-stable) (gh)
  dynamo/test_interop.py::InteropTests::test_trace_fn
• trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-stable) (gh)
  test_jit_fuser_te.py::TestTEFuserDynamic::test_where_ops
• trunk / macos-12-py3-arm64-mps / test (mps, 1, 1, macos-m1-stable) (gh)
  test_mps.py::TestAdvancedIndexing::test_nonzero_astuple_out
• trunk / win-vs2019-cpu-py3 / test (default, 1, 3, windows.4xlarge.nonephemeral) (gh)
  test_quantization.py::TestQuantizeDynamicJitOps::test_linear
• trunk / win-vs2019-cpu-py3 / test (default, 2, 3, windows.4xlarge.nonephemeral) (gh)
  test_jit_fuser_te.py::TestTEFuserDynamic::test_where_ops
• trunk / win-vs2019-cpu-py3 / test (default, 3, 3, windows.4xlarge.nonephemeral) (gh)
  test_ops_jit.py::TestJitCPU::test_variant_consistency_jit__chunk_cat_cpu_float32

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
  RuntimeError: run_func_( container_handle_, input_handles.data(), input_handles.size(), output_handles.data(), output_handles.size(), cuda_stream_handle, proxy_executor_handle_) API call failed at /var/lib/jenkins/workspace/torch/csrc/inductor/aoti_runner/model_container_runner.cpp, line 75
• pull / linux-jammy-py3.8-gcc11 / test (docs_test, 1, 1, linux.2xlarge) (gh) (similar failure)
  Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[tugsbayasgalan]

@tugsbayasgalan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[tugsbayasgalan]

@tugsbayasgalan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[tugsbayasgalan]

@tugsbayasgalan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[tugsbayasgalan]

@tugsbayasgalan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D56440612