[DO NOT MERGE] Reverting PR#71710 to test the performance impact with torchbench #72131

xuzhao9 · 2022-02-01T19:46:36Z

This PR is to test the performance improvement of PR #71710

//Convolution RUN_TORCHBENCH: alexnet, mnasnet1_0, mobilenet_v2, mobilenet_v3_large, resnet18, resnet50, squeezenet1_1, resnext50_32x4d, shufflenet_v2_x1_0, vgg16

RUN_TORCHBENCH: hf_Bert, hf_GPT2, hf_Bart, hf_BigBird, hf_DistilBert, hf_GPT2, hf_Longformer, hf_Reformer, hf_T5

pytorch-bot · 2022-02-01T19:46:40Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/b47ff28b287fab1ab19295ba15c23cd1bdc3d8d3/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default
Add ciflow labels to this PR to trigger more builds:

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-binary-conda	`ciflow/binaries`, `ciflow/binaries_conda`, `ciflow/default`	✅ triggered
linux-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
linux-binary-manywheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
linux-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/trunk`, `ciflow/xla`	✅ triggered
linux-docs	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/docs`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-vulkan-bionic-py3.7-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-cuda11.3-py3.7-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-build	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
linux-xenial-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/trunk`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/trunk`, `ciflow/win`	✅ triggered
windows-binary-libtorch-cxx11-abi	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-libtorch-pre-cxx11	`ciflow/binaries`, `ciflow/binaries_libtorch`, `ciflow/default`	✅ triggered
windows-binary-wheel	`ciflow/binaries`, `ciflow/binaries_wheel`, `ciflow/default`	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
docker-builds	`ciflow/all`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-custom-ops	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-arm64-metal	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
ios-12-5-1-x86-64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`, `ciflow/trunk`	🚫 skipped
linux-bionic-rocm4.5-py3.7	`ciflow/linux`, `ciflow/rocm`	🚫 skipped
linux-docs-push	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
linux-xenial-cuda11.3-py3.7-gcc7-no-ops	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-arm64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
macos-11-py3-x86-64	`ciflow/all`, `ciflow/macos`, `ciflow/trunk`	🚫 skipped
parallelnative-linux-xenial-py3.7-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped
periodic-libtorch-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-bionic-cuda11.5-py3.7-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`, `ciflow/slow`, `ciflow/slow-gradcheck`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.7-gcc7-debug	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
periodic-win-vs2019-cuda11.5-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-build	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/linux`, `ciflow/trunk`	🚫 skipped

facebook-github-bot · 2022-02-01T19:46:42Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/72131
📄 Preview docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 1aeb1d1 (more details on the Dr. CI page):

17/17 failures introduced in this PR

🕵️ 17 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

linux-bionic-py3.7-clang9 / test (noarch, 1, 1, linux.2xlarge) (1/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:27:28.2734060Z RuntimeError: distributions/test_distributions failed!

2022-03-11T23:27:27.8504201Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestKL-20220311232651.xml
2022-03-11T23:27:27.8508389Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestLazyLogitsInitialization-20220311232651.xml
2022-03-11T23:27:27.8520531Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestNumericalStability-20220311232651.xml
2022-03-11T23:27:27.8530563Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestRsample-20220311232651.xml
2022-03-11T23:27:27.8536547Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestValidation-20220311232651.xml
2022-03-11T23:27:28.2728694Z Traceback (most recent call last):
2022-03-11T23:27:28.2728962Z   File "test/run_test.py", line 1101, in <module>
2022-03-11T23:27:28.2730789Z     main()
2022-03-11T23:27:28.2731375Z   File "test/run_test.py", line 1079, in main
2022-03-11T23:27:28.2733607Z     raise RuntimeError(err_message)
2022-03-11T23:27:28.2734060Z RuntimeError: distributions/test_distributions failed!
2022-03-11T23:27:28.4581374Z 
2022-03-11T23:27:28.4581700Z real	0m41.364s
2022-03-11T23:27:28.4582035Z user	1m32.200s
2022-03-11T23:27:28.4582327Z sys	0m4.343s
2022-03-11T23:27:28.4582631Z + cleanup
2022-03-11T23:27:28.4582850Z + retcode=1
2022-03-11T23:27:28.4582991Z + set +x
2022-03-11T23:27:28.4626912Z ##[error]Process completed with exit code 1.
2022-03-11T23:27:28.4687257Z ##[group]Run # Ensure the working directory gets chowned back to the current user
2022-03-11T23:27:28.4687610Z �[36;1m# Ensure the working directory gets chowned back to the current user�[0m

linux-xenial-py3.7-gcc7 / test (default, 2, 2, linux.2xlarge) (2/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:48:09.4604716Z RuntimeError: test_spectral_ops failed!

2022-03-11T23:48:09.1407150Z FAILED (errors=1, skipped=2, expected failures=3)
2022-03-11T23:48:09.1407333Z 
2022-03-11T23:48:09.1407440Z Generating XML reports...
2022-03-11T23:48:09.1587877Z Generated XML report: test-reports/python-unittest/test_spectral_ops/TEST-TestFFTCPU-20220311234804.xml
2022-03-11T23:48:09.1608639Z Generated XML report: test-reports/python-unittest/test_spectral_ops/TEST-TestFFTDocExamplesCPU-20220311234804.xml
2022-03-11T23:48:09.4599261Z Traceback (most recent call last):
2022-03-11T23:48:09.4599525Z   File "test/run_test.py", line 1101, in <module>
2022-03-11T23:48:09.4601822Z     main()
2022-03-11T23:48:09.4602015Z   File "test/run_test.py", line 1079, in main
2022-03-11T23:48:09.4604460Z     raise RuntimeError(err_message)
2022-03-11T23:48:09.4604716Z RuntimeError: test_spectral_ops failed!
2022-03-11T23:48:09.6818286Z + cleanup
2022-03-11T23:48:09.6818512Z + retcode=1
2022-03-11T23:48:09.6818667Z + set +x
2022-03-11T23:48:09.6862635Z ##[error]Process completed with exit code 1.
2022-03-11T23:48:09.6897201Z ##[group]Run # Ensure the working directory gets chowned back to the current user
2022-03-11T23:48:09.6907559Z �[36;1m# Ensure the working directory gets chowned back to the current user�[0m
2022-03-11T23:48:09.6907915Z �[36;1mdocker run --rm -v "$(pwd)":/v -w /v "${ALPINE_IMAGE}" chown -R "$(id -u):$(id -g)" .�[0m
2022-03-11T23:48:09.6926037Z shell: /usr/bin/bash -e {0}
2022-03-11T23:48:09.6926230Z env:
2022-03-11T23:48:09.6926449Z   BUILD_ENVIRONMENT: linux-xenial-py3.7-gcc7

linux-bionic-rocm4.5-py3.7 / test (default, 1, 2, linux.rocm.gpu) (3/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-12T00:16:40.8284706Z RuntimeError: distributions/test_distributions failed!

2022-03-12T00:16:39.0929079Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestKL-20220312001604.xml
2022-03-12T00:16:39.0932277Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestLazyLogitsInitialization-20220312001604.xml
2022-03-12T00:16:39.0946185Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestNumericalStability-20220312001604.xml
2022-03-12T00:16:39.0956204Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestRsample-20220312001604.xml
2022-03-12T00:16:39.0961939Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestValidation-20220312001604.xml
2022-03-12T00:16:40.8269235Z Traceback (most recent call last):
2022-03-12T00:16:40.8270041Z   File "test/run_test.py", line 1101, in <module>
2022-03-12T00:16:40.8276779Z     main()
2022-03-12T00:16:40.8277426Z   File "test/run_test.py", line 1079, in main
2022-03-12T00:16:40.8283841Z     raise RuntimeError(err_message)
2022-03-12T00:16:40.8284706Z RuntimeError: distributions/test_distributions failed!
2022-03-12T00:16:42.4618641Z 
2022-03-12T00:16:42.4619168Z real	0m45.586s
2022-03-12T00:16:42.4619740Z user	2m35.866s
2022-03-12T00:16:42.4620251Z sys	0m18.269s
2022-03-12T00:16:42.4620820Z + cleanup
2022-03-12T00:16:42.4621308Z + retcode=1
2022-03-12T00:16:42.4621804Z + set +x
2022-03-12T00:16:42.4730231Z ##[error]Process completed with exit code 1.
2022-03-12T00:16:42.4791917Z ##[group]Run python3 -m pip install junitparser==2.1.1 rich==10.9.0
2022-03-12T00:16:42.4792585Z �[36;1mpython3 -m pip install junitparser==2.1.1 rich==10.9.0�[0m

linux-xenial-py3.7-clang7-onnx / test (default, 1, 2, linux.2xlarge) (4/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-12T00:00:58.8778857Z AttributeError: 'T...ls' object has no attribute 'onnx_shape_inference'

2022-03-12T00:00:58.8773732Z  �[36mtest/onnx/test_models.py�[0m::TestModels.test_srresnet�[0m �[34ms�[0m             �[31m37% �[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[31m▊�[0m�[40m�[31m      �[0m
2022-03-12T00:00:58.8774150Z 
2022-03-12T00:00:58.8774493Z ――――――――――――――――――――――― TestModels.test_super_resolution ―――――――――――――――――――――――
2022-03-12T00:00:58.8774886Z Traceback (most recent call last):
2022-03-12T00:00:58.8775341Z   File "/var/lib/jenkins/workspace/test/onnx/test_models.py", line 102, in test_super_resolution
2022-03-12T00:00:58.8775993Z     self.exportTest(toC(SuperResolutionNet(upscale_factor=3)), toC(x), atol=1e-6)
2022-03-12T00:00:58.8776536Z   File "/var/lib/jenkins/workspace/test/onnx/test_models_onnxruntime.py", line 17, in exportTest
2022-03-12T00:00:58.8776991Z     input=inputs, rtol=rtol, atol=atol)
2022-03-12T00:00:58.8777458Z   File "/var/lib/jenkins/workspace/test/onnx/test_pytorch_onnx_onnxruntime.py", line 171, in run_model_test
2022-03-12T00:00:58.8777956Z     onnx_shape_inference=self.onnx_shape_inference)
2022-03-12T00:00:58.8778857Z AttributeError: 'TestModels' object has no attribute 'onnx_shape_inference'
2022-03-12T00:00:58.8779143Z 
2022-03-12T00:00:58.8779150Z 
2022-03-12T00:00:58.8780278Z 
2022-03-12T00:00:58.8798704Z  �[36mtest/onnx/test_models.py�[0m::TestModels.test_super_resolution�[0m �[31m⨯�[0m     �[31m37% �[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[31m▊�[0m�[40m�[31m      �[0m
2022-03-12T00:00:58.8799100Z 
2022-03-12T00:00:58.8803342Z 
2022-03-12T00:00:58.8818865Z  �[36mtest/onnx/test_models.py�[0m::TestModels.test_vgg16�[0m �[34ms�[0m                �[31m37% �[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[31m▊�[0m�[40m�[31m      �[0m
2022-03-12T00:00:58.8819217Z 
2022-03-12T00:00:58.8822825Z 
2022-03-12T00:00:58.8837421Z  �[36mtest/onnx/test_models.py�[0m::TestModels.test_vgg16_bn�[0m �[34ms�[0m             �[31m37% �[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[32m█�[0m�[40m�[31m▊�[0m�[40m�[31m      �[0m

pytorch-xla-linux-bionic-py3.7-clang8 / build (5/17)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:03:00.5337593Z �[31mFAILED: �[0mc.../torch/csrc/distributed/c10d/ProcessGroupMPI.cpp.o

2022-03-11T23:02:54.6203106Z [4286/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupGloo.cpp.o�[K
2022-03-11T23:02:54.6204639Z [4287/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/comm.cpp.o�[K
2022-03-11T23:02:54.7256434Z [4287/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp.o�[K
2022-03-11T23:02:54.7257430Z [4288/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/FileStore.cpp.o�[K
2022-03-11T23:02:56.8788059Z [4288/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupWrapper.cpp.o�[K
2022-03-11T23:02:56.8789806Z [4289/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/serialization/export_module.cpp.o�[K
2022-03-11T23:02:58.4701293Z [4289/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/sequence_num.cpp.o�[K
2022-03-11T23:02:58.4702235Z [4290/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/PrefixStore.cpp.o�[K
2022-03-11T23:03:00.5335037Z [4290/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/Store.cpp.o�[K
2022-03-11T23:03:00.5336380Z [4291/4894] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp.o�[K
2022-03-11T23:03:00.5337593Z �[31mFAILED: �[0mcaffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp.o 
2022-03-11T23:03:00.5359211Z /opt/cargo/bin/sccache /usr/bin/clang++ -DADD_BREAKPAD_SIGNAL_HANDLER -DAT_PER_OPERATOR_HEADERS -DCPUINFO_SUPPORTED_PLATFORM=1 -DFMT_HEADER_ONLY=1 -DFXDIV_USE_INLINE_ASSEMBLY=0 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_C10D_GLOO -DUSE_C10D_MPI -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS -I/var/lib/jenkins/workspace/build/aten/src -I/var/lib/jenkins/workspace/aten/src -I/var/lib/jenkins/workspace/build -I/var/lib/jenkins/workspace -I/var/lib/jenkins/workspace/cmake/../third_party/benchmark/include -I/var/lib/jenkins/workspace/third_party/onnx -I/var/lib/jenkins/workspace/build/third_party/onnx -I/var/lib/jenkins/workspace/third_party/foxi -I/var/lib/jenkins/workspace/build/third_party/foxi -I/var/lib/jenkins/workspace/torch/csrc/api -I/var/lib/jenkins/workspace/torch/csrc/api/include -I/var/lib/jenkins/workspace/caffe2/aten/src/TH -I/var/lib/jenkins/workspace/build/caffe2/aten/src/TH -I/var/lib/jenkins/workspace/build/caffe2/aten/src -I/var/lib/jenkins/workspace/caffe2/../third_party -I/var/lib/jenkins/workspace/caffe2/../third_party/breakpad/src -I/var/lib/jenkins/workspace/build/caffe2/../aten/src -I/var/lib/jenkins/workspace/build/caffe2/../aten/src/ATen -I/var/lib/jenkins/workspace/torch/csrc -I/var/lib/jenkins/workspace/third_party/miniz-2.0.8 -I/var/lib/jenkins/workspace/third_party/kineto/libkineto/include -I/var/lib/jenkins/workspace/third_party/kineto/libkineto/src -I/var/lib/jenkins/workspace/torch/csrc/distributed -I/var/lib/jenkins/workspace/aten/../third_party/catch/single_include -I/var/lib/jenkins/workspace/aten/src/ATen/.. -I/var/lib/jenkins/workspace/build/caffe2/aten/src/ATen -I/var/lib/jenkins/workspace/third_party/FXdiv/include -I/var/lib/jenkins/workspace/c10/.. -I/var/lib/jenkins/workspace/third_party/pthreadpool/include -I/var/lib/jenkins/workspace/third_party/cpuinfo/include -I/var/lib/jenkins/workspace/third_party/QNNPACK/include -I/var/lib/jenkins/workspace/aten/src/ATen/native/quantized/cpu/qnnpack/include -I/var/lib/jenkins/workspace/aten/src/ATen/native/quantized/cpu/qnnpack/src -I/var/lib/jenkins/workspace/third_party/cpuinfo/deps/clog/include -I/var/lib/jenkins/workspace/third_party/NNPACK/include -I/var/lib/jenkins/workspace/third_party/fbgemm/include -I/var/lib/jenkins/workspace/third_party/fbgemm -I/var/lib/jenkins/workspace/third_party/fbgemm/third_party/asmjit/src -I/var/lib/jenkins/workspace/third_party/FP16/include -I/var/lib/jenkins/workspace/third_party/tensorpipe -I/var/lib/jenkins/workspace/build/third_party/tensorpipe -I/var/lib/jenkins/workspace/third_party/tensorpipe/third_party/libnop/include -I/var/lib/jenkins/workspace/third_party/fmt/include -I/var/lib/jenkins/workspace/build/third_party/ideep/mkl-dnn/third_party/oneDNN/include -I/var/lib/jenkins/workspace/third_party/ideep/mkl-dnn/third_party/oneDNN/src/../include -I/var/lib/jenkins/workspace/third_party/flatbuffers/include -isystem /var/lib/jenkins/workspace/build/third_party/gloo -isystem /var/lib/jenkins/workspace/cmake/../third_party/gloo -isystem /var/lib/jenkins/workspace/cmake/../third_party/googletest/googlemock/include -isystem /var/lib/jenkins/workspace/cmake/../third_party/googletest/googletest/include -isystem /var/lib/jenkins/workspace/third_party/protobuf/src -isystem /opt/conda/include -isystem /var/lib/jenkins/workspace/third_party/gemmlowp -isystem /var/lib/jenkins/workspace/third_party/neon2sse -isystem /var/lib/jenkins/workspace/third_party/XNNPACK/include -isystem /var/lib/jenkins/workspace/cmake/../third_party/eigen -isystem /opt/conda/include/python3.7m -isystem /opt/conda/lib/python3.7/site-packages/numpy/core/include -isystem /var/lib/jenkins/workspace/cmake/../third_party/pybind11/include -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -isystem /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -isystem /usr/lib/x86_64-linux-gnu/openmpi/include -isystem /var/lib/jenkins/workspace/third_party/ideep/mkl-dnn/third_party/oneDNN/include -isystem /var/lib/jenkins/workspace/third_party/ideep/include -isystem /var/lib/jenkins/workspace/build/include -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp=libomp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-un
2022-03-11T23:03:00.5373193Z �[1m/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp:698:10: �[0m�[0;1;31merror: �[0m�[1mlambda capture 'opts' is not used [-Werror,-Wunused-lambda-capture]�[0m
2022-03-11T23:03:00.5374080Z         [opts, this](std::unique_ptr<WorkEntry>& entry) {
2022-03-11T23:03:00.5374521Z �[0;1;32m         ^~~~~
2022-03-11T23:03:00.5378519Z �[0m�[1m/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp:725:10: �[0m�[0;1;31merror: �[0m�[1mlambda capture 'opts' is not used [-Werror,-Wunused-lambda-capture]�[0m
2022-03-11T23:03:00.5379238Z         [opts, this, inputSplitSizes, outputSplitSizes](
2022-03-11T23:03:00.5379707Z �[0;1;32m         ^~~~~
2022-03-11T23:03:00.5441373Z �[0m�[1m/var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp:772:8: �[0m�[0;1;31merror: �[0m�[1mlambda capture 'opts' is not used [-Werror,-Wunused-lambda-capture]�[0m
2022-03-11T23:03:00.5442161Z       [opts, this](std::unique_ptr<WorkEntry>& entry) {
2022-03-11T23:03:00.5442585Z �[0;1;32m       ^~~~~

linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (6/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:19:54.8375810Z RuntimeError:

2022-03-11T23:19:54.3289559Z Author: PyTorch Team
2022-03-11T23:19:54.3289842Z Author-email: packages@pytorch.org
2022-03-11T23:19:54.3290064Z License: BSD-3
2022-03-11T23:19:54.3290339Z Location: /opt/conda/lib/python3.7/site-packages
2022-03-11T23:19:54.3290617Z Requires: typing-extensions
2022-03-11T23:19:54.3290824Z Required-by: 
2022-03-11T23:19:54.3614451Z + python check_forward_backward_compatibility.py --existing-schemas nightly_schemas.txt
2022-03-11T23:19:54.8375058Z Traceback (most recent call last):
2022-03-11T23:19:54.8375354Z   File "check_forward_backward_compatibility.py", line 260, in <module>
2022-03-11T23:19:54.8375632Z     s = parse_schema(line.strip())
2022-03-11T23:19:54.8375810Z RuntimeError: 
2022-03-11T23:19:54.8376065Z Unknown custom class type profiler._RecordFunction. Please ensure it is registered.:
2022-03-11T23:19:54.8376629Z profiler::_record_function_exit._RecordFunction(__torch__.torch.classes.profiler._RecordFunction _0) -> ()
2022-03-11T23:19:54.8377016Z                                                                                  ~~~~~~~~~~~~~~~ <--- HERE
2022-03-11T23:19:54.8377149Z 
2022-03-11T23:19:54.9101417Z + cleanup
2022-03-11T23:19:54.9101688Z + retcode=1
2022-03-11T23:19:54.9101919Z + set +x
2022-03-11T23:19:54.9138795Z ##[error]Process completed with exit code 1.
2022-03-11T23:19:54.9167102Z ##[group]Run # Ensure the working directory gets chowned back to the current user
2022-03-11T23:19:54.9167513Z �[36;1m# Ensure the working directory gets chowned back to the current user�[0m

linux-bionic-py3.7-clang9 / test (default, 2, 2, linux.2xlarge) (7/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:52:23.3288107Z AssertionError: can only test a child process

2022-03-11T23:52:23.1905182Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-11T23:52:23.1905804Z AssertionError: can only test a child process
2022-03-11T23:52:23.3256770Z Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7fc156fb97a0>
2022-03-11T23:52:23.3257374Z Traceback (most recent call last):
2022-03-11T23:52:23.3258022Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1337, in __del__
2022-03-11T23:52:23.3260969Z     self._shutdown_workers()
2022-03-11T23:52:23.3261477Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1329, in _shutdown_workers
2022-03-11T23:52:23.3285929Z     if w.is_alive():
2022-03-11T23:52:23.3286656Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-03-11T23:52:23.3287686Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-11T23:52:23.3288107Z AssertionError: can only test a child process
2022-03-11T23:52:23.9571079Z ok (0.798s)
2022-03-11T23:52:23.9595864Z   test_iterabledataset_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.002s)
2022-03-11T23:52:25.3156182Z   test_large_sampler_indices (__main__.TestDataLoaderPersistentWorkers) ... ok (1.356s)
2022-03-11T23:52:25.3227538Z   test_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.007s)
2022-03-11T23:52:26.7162827Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... ok (1.393s)
2022-03-11T23:52:29.6278538Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... ok (2.911s)
2022-03-11T23:52:30.3761961Z   test_no_segfault (__main__.TestDataLoaderPersistentWorkers) ... ok (0.748s)
2022-03-11T23:52:30.3788022Z   test_numpy (__main__.TestDataLoaderPersistentWorkers) ... ok (0.003s)
2022-03-11T23:52:30.3831888Z   test_numpy_gen_state (__main__.TestDataLoaderPersistentWorkers) ... ok (0.004s)
2022-03-11T23:52:30.3851867Z   test_numpy_scalars (__main__.TestDataLoaderPersistentWorkers) ... ok (0.002s)

linux-bionic-py3.7-clang9 / test (default, 1, 2, linux.2xlarge) (8/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:55:09.0097352Z RuntimeError: distributions/test_distributions failed!

2022-03-11T23:55:08.5424457Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestKL-20220311235429.xml
2022-03-11T23:55:08.5427399Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestLazyLogitsInitialization-20220311235429.xml
2022-03-11T23:55:08.5439682Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestNumericalStability-20220311235429.xml
2022-03-11T23:55:08.5447497Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestRsample-20220311235429.xml
2022-03-11T23:55:08.5452728Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestValidation-20220311235429.xml
2022-03-11T23:55:09.0091573Z Traceback (most recent call last):
2022-03-11T23:55:09.0091966Z   File "test/run_test.py", line 1101, in <module>
2022-03-11T23:55:09.0093672Z     main()
2022-03-11T23:55:09.0094016Z   File "test/run_test.py", line 1079, in main
2022-03-11T23:55:09.0096933Z     raise RuntimeError(err_message)
2022-03-11T23:55:09.0097352Z RuntimeError: distributions/test_distributions failed!
2022-03-11T23:55:09.2603845Z 
2022-03-11T23:55:09.2604198Z real	27m53.275s
2022-03-11T23:55:09.2604407Z user	63m17.751s
2022-03-11T23:55:09.2604563Z sys	2m38.261s
2022-03-11T23:55:09.2604732Z + cleanup
2022-03-11T23:55:09.2604871Z + retcode=1
2022-03-11T23:55:09.2605022Z + set +x
2022-03-11T23:55:09.2656337Z ##[error]Process completed with exit code 1.
2022-03-11T23:55:09.2688946Z ##[group]Run # Ensure the working directory gets chowned back to the current user
2022-03-11T23:55:09.2689302Z �[36;1m# Ensure the working directory gets chowned back to the current user�[0m

linux-xenial-py3.7-gcc5.4 / test (default, 2, 2, linux.2xlarge) (9/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:46:06.8114401Z RuntimeError: distributions/test_distributions failed!

2022-03-11T23:46:06.4061105Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestKL-20220311234528.xml
2022-03-11T23:46:06.4084285Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestLazyLogitsInitialization-20220311234528.xml
2022-03-11T23:46:06.4093699Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestNumericalStability-20220311234528.xml
2022-03-11T23:46:06.4100800Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestRsample-20220311234528.xml
2022-03-11T23:46:06.4105153Z Generated XML report: test-reports/python-unittest/distributions.test_distributions/TEST-TestValidation-20220311234528.xml
2022-03-11T23:46:06.8109648Z Traceback (most recent call last):
2022-03-11T23:46:06.8109927Z   File "test/run_test.py", line 1101, in <module>
2022-03-11T23:46:06.8111784Z     main()
2022-03-11T23:46:06.8112022Z   File "test/run_test.py", line 1079, in main
2022-03-11T23:46:06.8114092Z     raise RuntimeError(err_message)
2022-03-11T23:46:06.8114401Z RuntimeError: distributions/test_distributions failed!
2022-03-11T23:46:07.0056692Z + cleanup
2022-03-11T23:46:07.0057003Z + retcode=1
2022-03-11T23:46:07.0057161Z + set +x
2022-03-11T23:46:07.0098469Z ##[error]Process completed with exit code 1.
2022-03-11T23:46:07.0169210Z ##[group]Run # Ensure the working directory gets chowned back to the current user
2022-03-11T23:46:07.0169568Z �[36;1m# Ensure the working directory gets chowned back to the current user�[0m
2022-03-11T23:46:07.0169882Z �[36;1mdocker run --rm -v "$(pwd)":/v -w /v "${ALPINE_IMAGE}" chown -R "$(id -u):$(id -g)" .�[0m
2022-03-11T23:46:07.0509299Z shell: /usr/bin/bash -e {0}
2022-03-11T23:46:07.0509480Z env:
2022-03-11T23:46:07.0509679Z   BUILD_ENVIRONMENT: linux-xenial-py3.7-gcc5.4

linux-bionic-rocm4.5-py3.7 / test (default, 2, 2, linux.rocm.gpu) (10/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-12T00:17:19.7328152Z AssertionError: can only test a child process

2022-03-12T00:17:19.7189572Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-12T00:17:19.7191119Z AssertionError: can only test a child process
2022-03-12T00:17:19.7312445Z Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f614bb91a70>
2022-03-12T00:17:19.7313493Z Traceback (most recent call last):
2022-03-12T00:17:19.7315025Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1337, in __del__
2022-03-12T00:17:19.7319823Z     self._shutdown_workers()
2022-03-12T00:17:19.7321530Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1329, in _shutdown_workers
2022-03-12T00:17:19.7324631Z     if w.is_alive():
2022-03-12T00:17:19.7325562Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-03-12T00:17:19.7327051Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-12T00:17:19.7328152Z AssertionError: can only test a child process
2022-03-12T00:17:20.4614674Z ok (0.774s)
2022-03-12T00:17:20.4680995Z   test_iterabledataset_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.006s)
2022-03-12T00:17:23.9983889Z   test_large_sampler_indices (__main__.TestDataLoaderPersistentWorkers) ... ok (3.530s)
2022-03-12T00:17:24.0148627Z   test_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.016s)
2022-03-12T00:17:27.2789790Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... ok (3.264s)
2022-03-12T00:17:29.2791142Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:17:29.5050946Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:17:29.5202799Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:17:33.3611911Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:17:33.3975569Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

win-vs2019-cuda11.3-py3 / build (11/17)

Step: "Cleanup build-results and workspaces" (full log | diagnosis details | 🔁 rerun)

2022-03-11T22:44:58.9465221Z ls: cannot access ...d/win_tmp/ci_scripts/*': No such file or directory

2022-03-11T22:44:58.7328250Z ++ cygpath -w /c/actions-runner/_work/pytorch/pytorch/build/win_tmp
2022-03-11T22:44:58.7750716Z + TMP_DIR_WIN='C:\actions-runner\_work\pytorch\pytorch\build\win_tmp'
2022-03-11T22:44:58.7751232Z + export TMP_DIR_WIN
2022-03-11T22:44:58.7751658Z + export PYTORCH_FINAL_PACKAGE_DIR=/c/1971309294/build-results/
2022-03-11T22:44:58.7752189Z + PYTORCH_FINAL_PACKAGE_DIR=/c/1971309294/build-results/
2022-03-11T22:44:58.7752607Z + [[ -n /c/1971309294/build-results/ ]]
2022-03-11T22:44:58.7752992Z + mkdir -p /c/1971309294/build-results/
2022-03-11T22:44:58.8371277Z + CI_SCRIPTS_DIR=/c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts
2022-03-11T22:44:58.8371964Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts
2022-03-11T22:44:58.8555044Z ++ ls '/c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts/*'
2022-03-11T22:44:58.9465221Z ls: cannot access '/c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts/*': No such file or directory
2022-03-11T22:44:58.9467918Z + '[' -n '' ']'
2022-03-11T22:44:58.9468555Z + export SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/.jenkins/pytorch/win-test-helpers
2022-03-11T22:44:58.9469389Z + SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/.jenkins/pytorch/win-test-helpers
2022-03-11T22:44:58.9469902Z + set +ex
2022-03-11T22:45:08.9159749Z + /c/actions-runner/_work/pytorch/pytorch/.jenkins/pytorch/win-test-helpers/build_pytorch.bat
2022-03-11T22:45:08.9409478Z 
2022-03-11T22:45:08.9410313Z C:\actions-runner\_work\pytorch\pytorch>if "" == "1" (set BUILD_TYPE=debug )  ELSE (set BUILD_TYPE=release ) 
2022-03-11T22:45:08.9413605Z 
2022-03-11T22:45:08.9416142Z C:\actions-runner\_work\pytorch\pytorch>set PATH=C:\Program Files\CMake\bin;C:\Program Files\7-Zip;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Amazon\AWSCLI;C:\Program Files\Amazon\AWSCLI\bin;C:\Users\runneruser\AppData\Roaming\Python\Python310\Scripts;C:\actions-runner\_work\_tool\Python\3.10.2\x64\Scripts;C:\actions-runner\_work\_tool\Python\3.10.2\x64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Amazon\cfn-bootstrap;C:\ProgramData\chocolatey\bin;C:\Program Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Users\runneruser\AppData\Local\Microsoft\WindowsApps 
2022-03-11T22:45:08.9418507Z

linux-xenial-py3.7-gcc5.4 / test (default, 1, 2, linux.2xlarge) (12/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:47:05.6416026Z AssertionError: can only test a child process

2022-03-11T23:47:05.6389243Z     if w.is_alive():
2022-03-11T23:47:05.6389795Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-03-11T23:47:05.6391836Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-11T23:47:05.6392223Z AssertionError: can only test a child process
2022-03-11T23:47:05.6408326Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1337, in __del__
2022-03-11T23:47:05.6411547Z     self._shutdown_workers()
2022-03-11T23:47:05.6412002Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1329, in _shutdown_workers
2022-03-11T23:47:05.6414305Z     if w.is_alive():
2022-03-11T23:47:05.6414635Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-03-11T23:47:05.6415622Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-11T23:47:05.6416026Z AssertionError: can only test a child process
2022-03-11T23:47:06.2240220Z ok (0.627s)
2022-03-11T23:47:06.2263121Z   test_iterabledataset_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.002s)
2022-03-11T23:47:07.4961101Z   test_large_sampler_indices (__main__.TestDataLoaderPersistentWorkers) ... ok (1.270s)
2022-03-11T23:47:07.5035228Z   test_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.007s)
2022-03-11T23:47:08.7096002Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... ok (1.206s)
2022-03-11T23:47:11.3765640Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... ok (2.667s)
2022-03-11T23:47:12.0278389Z   test_no_segfault (__main__.TestDataLoaderPersistentWorkers) ... ok (0.651s)
2022-03-11T23:47:12.0304225Z   test_numpy (__main__.TestDataLoaderPersistentWorkers) ... ok (0.003s)
2022-03-11T23:47:12.0347580Z   test_numpy_gen_state (__main__.TestDataLoaderPersistentWorkers) ... ok (0.004s)
2022-03-11T23:47:12.0366604Z   test_numpy_scalars (__main__.TestDataLoaderPersistentWorkers) ... ok (0.002s)

linux-xenial-cuda11.3-py3.7-gcc7 / test (default, 1, 2, linux.4xlarge.nvidia.gpu) (13/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-12T00:44:04.2372065Z AssertionError: can only test a child process

2022-03-12T00:44:04.2077352Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-12T00:44:04.2077934Z AssertionError: can only test a child process
2022-03-12T00:44:04.2361166Z Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f934a63a8c0>
2022-03-12T00:44:04.2361592Z Traceback (most recent call last):
2022-03-12T00:44:04.2362236Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1337, in __del__
2022-03-12T00:44:04.2367318Z     self._shutdown_workers()
2022-03-12T00:44:04.2367883Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1329, in _shutdown_workers
2022-03-12T00:44:04.2370849Z     if w.is_alive():
2022-03-12T00:44:04.2371185Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-03-12T00:44:04.2371701Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-12T00:44:04.2372065Z AssertionError: can only test a child process
2022-03-12T00:44:05.4998287Z ok (1.362s)
2022-03-12T00:44:05.5032579Z   test_iterabledataset_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.003s)
2022-03-12T00:44:07.4764087Z   test_large_sampler_indices (__main__.TestDataLoaderPersistentWorkers) ... ok (1.973s)
2022-03-12T00:44:07.4887015Z   test_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.012s)
2022-03-12T00:44:09.5543796Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... ok (2.065s)
2022-03-12T00:44:12.5021779Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:44:12.5022546Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:44:12.5025402Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:44:15.4589773Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-03-12T00:44:15.4590392Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

linux-xenial-py3.7-gcc7 / test (default, 1, 2, linux.2xlarge) (14/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:46:57.0445863Z AssertionError: can only test a child process

2022-03-11T23:46:56.9170626Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-11T23:46:56.9171087Z AssertionError: can only test a child process
2022-03-11T23:46:57.0437885Z Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f948a01e7a0>
2022-03-11T23:46:57.0438401Z Traceback (most recent call last):
2022-03-11T23:46:57.0439134Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1337, in __del__
2022-03-11T23:46:57.0441941Z     self._shutdown_workers()
2022-03-11T23:46:57.0442683Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1329, in _shutdown_workers
2022-03-11T23:46:57.0444302Z     if w.is_alive():
2022-03-11T23:46:57.0444728Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-03-11T23:46:57.0445402Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-03-11T23:46:57.0445863Z AssertionError: can only test a child process
2022-03-11T23:46:57.4896311Z ok (0.615s)
2022-03-11T23:46:57.4919113Z   test_iterabledataset_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.002s)
2022-03-11T23:46:58.7366863Z   test_large_sampler_indices (__main__.TestDataLoaderPersistentWorkers) ... ok (1.245s)
2022-03-11T23:46:58.7442808Z   test_len (__main__.TestDataLoaderPersistentWorkers) ... ok (0.008s)
2022-03-11T23:46:59.9405135Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... ok (1.196s)
2022-03-11T23:47:02.5969687Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... ok (2.656s)
2022-03-11T23:47:03.2392431Z   test_no_segfault (__main__.TestDataLoaderPersistentWorkers) ... ok (0.642s)
2022-03-11T23:47:03.2416936Z   test_numpy (__main__.TestDataLoaderPersistentWorkers) ... ok (0.003s)
2022-03-11T23:47:03.2459874Z   test_numpy_gen_state (__main__.TestDataLoaderPersistentWorkers) ... ok (0.004s)
2022-03-11T23:47:03.2479063Z   test_numpy_scalars (__main__.TestDataLoaderPersistentWorkers) ... ok (0.002s)

linux-xenial-py3.7-clang7-asan / test (default, 2, 3, linux.2xlarge) (15/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-12T00:40:48.6945408Z RuntimeError: test_spectral_ops failed!

2022-03-12T00:40:48.2150674Z FAILED (errors=1, skipped=2, expected failures=3)
2022-03-12T00:40:48.2150888Z 
2022-03-12T00:40:48.2150995Z Generating XML reports...
2022-03-12T00:40:48.2365982Z Generated XML report: test-reports/python-unittest/test_spectral_ops/TEST-TestFFTCPU-20220312004029.xml
2022-03-12T00:40:48.2391640Z Generated XML report: test-reports/python-unittest/test_spectral_ops/TEST-TestFFTDocExamplesCPU-20220312004029.xml
2022-03-12T00:40:48.6934369Z Traceback (most recent call last):
2022-03-12T00:40:48.6934648Z   File "test/run_test.py", line 1101, in <module>
2022-03-12T00:40:48.6939457Z     main()
2022-03-12T00:40:48.6939688Z   File "test/run_test.py", line 1079, in main
2022-03-12T00:40:48.6945120Z     raise RuntimeError(err_message)
2022-03-12T00:40:48.6945408Z RuntimeError: test_spectral_ops failed!
2022-03-12T00:40:49.0692041Z + cleanup
2022-03-12T00:40:49.0692271Z + retcode=1
2022-03-12T00:40:49.0724448Z + set +x
2022-03-12T00:40:49.0736996Z ##[error]Process completed with exit code 1.
2022-03-12T00:40:49.0787232Z ##[group]Run # Ensure the working directory gets chowned back to the current user
2022-03-12T00:40:49.0787595Z �[36;1m# Ensure the working directory gets chowned back to the current user�[0m
2022-03-12T00:40:49.0787910Z �[36;1mdocker run --rm -v "$(pwd)":/v -w /v "${ALPINE_IMAGE}" chown -R "$(id -u):$(id -g)" .�[0m
2022-03-12T00:40:49.1234775Z shell: /usr/bin/bash -e {0}
2022-03-12T00:40:49.1234961Z env:
2022-03-12T00:40:49.1235200Z   BUILD_ENVIRONMENT: linux-xenial-py3.7-clang7-asan

win-vs2019-cpu-py3 / build (16/17)

Step: "Cleanup build-results and workspaces" (full log | diagnosis details | 🔁 rerun)

2022-03-11T22:42:18.2738518Z ls: cannot access ...d/win_tmp/ci_scripts/*': No such file or directory

2022-03-11T22:42:17.9435720Z ++ cygpath -w /c/actions-runner/_work/pytorch/pytorch/build/win_tmp
2022-03-11T22:42:18.0667515Z + TMP_DIR_WIN='C:\actions-runner\_work\pytorch\pytorch\build\win_tmp'
2022-03-11T22:42:18.0668225Z + export TMP_DIR_WIN
2022-03-11T22:42:18.0668661Z + export PYTORCH_FINAL_PACKAGE_DIR=/c/1971309310/build-results/
2022-03-11T22:42:18.0669162Z + PYTORCH_FINAL_PACKAGE_DIR=/c/1971309310/build-results/
2022-03-11T22:42:18.0669589Z + [[ -n /c/1971309310/build-results/ ]]
2022-03-11T22:42:18.0669968Z + mkdir -p /c/1971309310/build-results/
2022-03-11T22:42:18.1354999Z + CI_SCRIPTS_DIR=/c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts
2022-03-11T22:42:18.1355741Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts
2022-03-11T22:42:18.1540350Z ++ ls '/c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts/*'
2022-03-11T22:42:18.2738518Z ls: cannot access '/c/actions-runner/_work/pytorch/pytorch/build/win_tmp/ci_scripts/*': No such file or directory
2022-03-11T22:42:18.2742123Z + '[' -n '' ']'
2022-03-11T22:42:18.2742782Z + export SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/.jenkins/pytorch/win-test-helpers
2022-03-11T22:42:18.2743642Z + SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/.jenkins/pytorch/win-test-helpers
2022-03-11T22:42:18.2744146Z + set +ex
2022-03-11T22:42:27.8357582Z + /c/actions-runner/_work/pytorch/pytorch/.jenkins/pytorch/win-test-helpers/build_pytorch.bat
2022-03-11T22:42:27.8616183Z 
2022-03-11T22:42:27.8617019Z C:\actions-runner\_work\pytorch\pytorch>if "" == "1" (set BUILD_TYPE=debug )  ELSE (set BUILD_TYPE=release ) 
2022-03-11T22:42:27.8619938Z 
2022-03-11T22:42:27.8622416Z C:\actions-runner\_work\pytorch\pytorch>set PATH=C:\Program Files\CMake\bin;C:\Program Files\7-Zip;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Amazon\AWSCLI;C:\Program Files\Amazon\AWSCLI\bin;C:\Users\runneruser\AppData\Roaming\Python\Python310\Scripts;C:\actions-runner\_work\_tool\Python\3.10.2\x64\Scripts;C:\actions-runner\_work\_tool\Python\3.10.2\x64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Amazon\cfn-bootstrap;C:\ProgramData\chocolatey\bin;C:\Program Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Users\runneruser\AppData\Local\Microsoft\WindowsApps 
2022-03-11T22:42:27.8625416Z

linux-xenial-py3.7-clang7-asan / test (default, 3, 3, linux.2xlarge) (17/17)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-03-11T23:37:27.6642787Z SUMMARY: Undefined.../jenkins/workspace/aten/src/ATen/Utils.cpp:20:3 in

2022-03-11T23:37:27.6103934Z     #10 0x559ad0d21801 in run_mod /tmp/build/80754af9/python_1627392990942/work/Python/pythonrun.c:1037
2022-03-11T23:37:27.6104490Z     #11 0x559ad0d2c7a9 in PyRun_StringFlags /tmp/build/80754af9/python_1627392990942/work/Python/pythonrun.c:961
2022-03-11T23:37:27.6105442Z     #12 0x559ad0d2c80b in PyRun_SimpleStringFlags /tmp/build/80754af9/python_1627392990942/work/Python/pythonrun.c:455
2022-03-11T23:37:27.6106548Z     #13 0x559ad0d2c908 in pymain_run_command /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:420
2022-03-11T23:37:27.6107207Z     #14 0x559ad0d2c908 in pymain_run_python /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:2907
2022-03-11T23:37:27.6107531Z     #15 0x559ad0d2c908 in pymain_main /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:3460
2022-03-11T23:37:27.6108626Z     #16 0x559ad0d2cccb in _Py_UnixMain /tmp/build/80754af9/python_1627392990942/work/Modules/main.c:3495
2022-03-11T23:37:27.6641729Z     #17 0x7f54099f283f in __libc_start_main /build/glibc-S7Ft5T/glibc-2.23/csu/../csu/libc-start.c:291
2022-03-11T23:37:27.6642327Z     #18 0x559ad0cd1554 in _start (/opt/conda/bin/python3.7+0x1d7554)
2022-03-11T23:37:27.6642485Z 
2022-03-11T23:37:27.6642787Z SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:20:3 in 
2022-03-11T23:37:27.6814260Z + retcode=1
2022-03-11T23:37:27.6814720Z + set -e
2022-03-11T23:37:27.6814967Z + return 1
2022-03-11T23:37:27.6818237Z + [[ linux-xenial-py3.7-clang7-asan-default == *-NO_AVX-* ]]
2022-03-11T23:37:27.6818712Z + [[ default == \n\o\g\p\u\_\N\O\_\A\V\X ]]
2022-03-11T23:37:27.6819273Z + [[ linux-xenial-py3.7-clang7-asan-default == *-NO_AVX2-* ]]
2022-03-11T23:37:27.6819566Z + [[ default == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-03-11T23:37:27.6819873Z + [[ linux-xenial-py3.7-clang7-asan-default == *-NO_AVX512-* ]]
2022-03-11T23:37:27.6820279Z + [[ default == \n\o\g\p\u\_\N\O\_\A\V\X\5\1\2 ]]
2022-03-11T23:37:27.6823751Z + [[ linux-xenial-py3.7-clang7-asan-default == *tbb* ]]

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

xuzhao9 · 2022-03-11T22:29:25Z

Reopen it for another test

Reverting PR#71710 to test the performance impact with torchbench

b47ff28

pytorch-bot bot added the ciflow/default label Feb 1, 2022

facebook-github-bot added the cla signed label Feb 1, 2022

Trigger TorchBench CI again

b0a6b3b

xuzhao9 closed this Feb 2, 2022

xuzhao9 deleted the xz9/test-pr-71710 branch February 2, 2022 04:13

xuzhao9 restored the xz9/test-pr-71710 branch March 11, 2022 22:29

xuzhao9 reopened this Mar 11, 2022

Trigger the benchmark ci again

1aeb1d1

xuzhao9 closed this Mar 14, 2022

xuzhao9 deleted the xz9/test-pr-71710 branch September 18, 2022 12:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] Reverting PR#71710 to test the performance impact with torchbench #72131

[DO NOT MERGE] Reverting PR#71710 to test the performance impact with torchbench #72131

xuzhao9 commented Feb 1, 2022 •

edited

pytorch-bot bot commented Feb 1, 2022

⚛️ CI Flow

facebook-github-bot commented Feb 1, 2022 •

edited

xuzhao9 commented Mar 11, 2022

[DO NOT MERGE] Reverting PR#71710 to test the performance impact with torchbench #72131

[DO NOT MERGE] Reverting PR#71710 to test the performance impact with torchbench #72131

Conversation

xuzhao9 commented Feb 1, 2022 • edited

pytorch-bot bot commented Feb 1, 2022

⚛️ CI Flow

facebook-github-bot commented Feb 1, 2022 • edited

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 17 new failures recognized by patterns

linux-bionic-py3.7-clang9 / test (noarch, 1, 1, linux.2xlarge) (1/17)

linux-xenial-py3.7-gcc7 / test (default, 2, 2, linux.2xlarge) (2/17)

linux-bionic-rocm4.5-py3.7 / test (default, 1, 2, linux.rocm.gpu) (3/17)

linux-xenial-py3.7-clang7-onnx / test (default, 1, 2, linux.2xlarge) (4/17)

pytorch-xla-linux-bionic-py3.7-clang8 / build (5/17)

linux-xenial-py3.7-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (6/17)

linux-bionic-py3.7-clang9 / test (default, 2, 2, linux.2xlarge) (7/17)

linux-bionic-py3.7-clang9 / test (default, 1, 2, linux.2xlarge) (8/17)

linux-xenial-py3.7-gcc5.4 / test (default, 2, 2, linux.2xlarge) (9/17)

linux-bionic-rocm4.5-py3.7 / test (default, 2, 2, linux.rocm.gpu) (10/17)

win-vs2019-cuda11.3-py3 / build (11/17)

linux-xenial-py3.7-gcc5.4 / test (default, 1, 2, linux.2xlarge) (12/17)

linux-xenial-cuda11.3-py3.7-gcc7 / test (default, 1, 2, linux.4xlarge.nvidia.gpu) (13/17)

linux-xenial-py3.7-gcc7 / test (default, 1, 2, linux.2xlarge) (14/17)

linux-xenial-py3.7-clang7-asan / test (default, 2, 3, linux.2xlarge) (15/17)

win-vs2019-cpu-py3 / build (16/17)

linux-xenial-py3.7-clang7-asan / test (default, 3, 3, linux.2xlarge) (17/17)

xuzhao9 commented Mar 11, 2022

xuzhao9 commented Feb 1, 2022 •

edited

facebook-github-bot commented Feb 1, 2022 •

edited