ONNX tests failing on master

[datumbox]

🐛 Bug

The latest pytorch nightly (1.10.0.dev20210825) causes the ONNX tests to fail. See #4313

test/test_onnx.py .....s........Fatal Python error: Segmentation fault

Current thread 0x00007ffbcdf02740 (most recent call first):
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/jit/_trace.py", line 1121 in wrapper
  File "/home/circleci/.local/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 68 in batched_nms
  File "/home/circleci/.local/lib/python3.7/site-packages/torchvision/models/detection/rpn.py", line 271 in filter_proposals
  File "/home/circleci/.local/lib/python3.7/site-packages/torchvision/models/detection/rpn.py", line 356 in forward
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1053 in _slow_forward
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1065 in _call_impl
  File "/home/circleci/.local/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 96 in forward
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1053 in _slow_forward
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1065 in _call_impl
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/jit/_trace.py", line 118 in wrapper
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/jit/_trace.py", line 132 in forward
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1065 in _call_impl
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/jit/_trace.py", line 1166 in _get_trace_graph
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/onnx/utils.py", line 366 in _trace_and_get_graph_from_model
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/onnx/utils.py", line 415 in _create_jit_graph
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/onnx/utils.py", line 464 in _model_to_graph
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/onnx/utils.py", line 695 in _export
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/onnx/utils.py", line 90 in export
  File "/home/circleci/.local/lib/python3.7/site-packages/torch/onnx/__init__.py", line 324 in export
  File "/home/circleci/project/test/test_onnx.py", line 40 in run_model
  File "/home/circleci/project/test/test_onnx.py", line 442 in test_mask_rcnn
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/circleci/.local/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/circleci/.local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163 in main
  File "test/test_onnx.py", line 509 in <module>

Received "segmentation fault" signal

cc @neginraoof, @jiafatom

[neginraoof]

Thanks @datumbox for reporting this. Do you know when the failure has started? Is it coming from today's nightly?

[datumbox]

@neginraoof Correct, this is from today's nightly. I update the description with the exact version that broke it.

[neginraoof]

Just to clarify, I confirm that the issue has started since yesterday (the nightly for 0824 works fine). We haven't had any updates merged into master from dev ONNX branch. I am going to confirm this with Mark Saroufim as well.

From an initial look at the issue, seems like removing the call to disable the profiler is fixing the segfault:
https://github.com/pytorch/vision/pull/4316/files

I am going to debug this to find more details.

[jiafatom]

There is a PR directly merged into master branch, but that is on 8/23/2021, pytorch/pytorch@172e5c7

[bertmaher]

I found an issue a couple days ago where ONNX tests segfault due to turning on and off the profiling executor (which isn’t really well supported). It seems likely to be the same issue here.

[bertmaher]

pytorch/pytorch@1787b90#diff-27c821d0bcd55a0d2feaebeb93b313c3d3e255659d698d41da69578522e4ff59

[neginraoof]

Yeah this sounds like the same issue. Though I'm seeing some other (unrelated) failures on the PR here:
#4316

[datumbox]

@neginraoof The other unrelated failures are being discussed at #4315