GetPythonFrames segfault

[JackCaoG]

🐛 Bug

torch::lazy::GetPythonFrames segfault

31│ std::vector<SourceLocation> GetPythonFrames() {
32│   std::vector<SourceLocation> frames;
33│   if (Py_IsInitialized()) {
34│     pybind11::gil_scoped_acquire gil;
35│     PyFrameObject* frame = PyEval_GetFrame();
36├>    Py_INCREF(frame);
37│     while (frame != nullptr) {

Thread 573 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff8c1ffb700 (LWP 112450)]
0x00007fffeacc52b2 in torch::lazy::GetPythonFrames () at ../torch/csrc/lazy/python/python_util.cpp:36
(gdb) p frame
$1 = (PyFrameObject *) 0x0

To Reproduce

XLA_IR_DEBUG=1 python ../test/test_nn.py TestNNDeviceTypeXLA.test_Dropout2d_xla

[JackCaoG]

I think the rootcause is

(gdb) p PyEval_GetFrame()
$4 = (PyFrameObject *) 0x0`

but I am not sure why PyEval_GetFrame would return an empty pointer.

[JackCaoG]

@miladm as I think you run into the same issue before.

[miladm]

Yeah, I filed this issue #3795 when I ran into a similar issue. In my case, I was testing the dynamic shape code plumbing when the code crashed. It some changes to (perhaps) pytorch after around July 20th caused the failure.

CC @Krovatkin for visibility.

[JackCaoG]

I added

    if (frame == nullptr) {
      return {};
    }

toGetPythonFrames in bypass this issue, at least we should not segfault. trying to verify if XLA_IR_DEBUG=1 works for most cases.

[JackCaoG]

We actually already have a test test/test_env_var_mapper.py to test that XLA_IR_DEBUG works.

(Pdb) xla_tensors_report
'Tensor: id=1, shape=f32[2]{0}, device=CPU:0, ir_nodes=1\nFrames:\n  test_xla_ir_debug (test/test_env_var_mapper.py:18)\n  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/case.py:628)\n  __call__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/case.py:676)\n  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:122)\n  __call__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:84)\n  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:122)\n  __call__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:84)\n  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/runner.py:176)\n  runTests (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/main.py:271)\n  __init__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/main.py:101)\n  <module> (test/test_env_var_mapper.py:28)\nIR {\n  %0 = f32[2]{0} xla::device_data(), location=test_xla_ir_debug@test_env_var_mapper.py:18, device=CPU:0, ROOT=0\n}\n\n\n'
(Pdb) print(xla_tensors_report)
Tensor: id=1, shape=f32[2]{0}, device=CPU:0, ir_nodes=1
Frames:
  test_xla_ir_debug (test/test_env_var_mapper.py:18)
  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/case.py:628)
  __call__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/case.py:676)
  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:122)
  __call__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:84)
  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:122)
  __call__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/suite.py:84)
  run (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/runner.py:176)
  runTests (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/main.py:271)
  __init__ (/root/anaconda3/envs/pytorch/lib/python3.7/unittest/main.py:101)
  <module> (test/test_env_var_mapper.py:28)
IR {
  %0 = f32[2]{0} xla::device_data(), location=test_xla_ir_debug@test_env_var_mapper.py:18, device=CPU:0, ROOT=0
}

I guess the bug here that in some case python frame will be empty, I will send a quick fix to pytorch.

[JackCaoG]

pytorch/pytorch#83643 should fix this issue.