Remote debugging can lead to core dump (Prop#283579)

[rsaim]

reproducer.py

import pyflyby
import sys

def mydebugger(*args, **kwargs):
    kwargs["wait_for_attach"] = True
    pyflyby.debugger(*args, **kwargs)

sys.excepthook = mydebugger

1/0

1. Run the reproducer above
2. Try to connect to the process using instructions displayed using the py script - py -D <PID>.
3. Send a SIGINT to the reproducer.py process

We get the following stacktrace.

[PYFLYBY] While running /u/raza/.PyCharmCE2019.1/config/scratche, ZeroDivisionError in <module> at /u/raza/reproducer.py:10
[PYFLYBY]
[PYFLYBY] Please run:
[PYFLYBY]     ssh -t testhost /usr/local/python/python-3.7/std/bin/py -d 16243
[PYFLYBY]
[PYFLYBY] As process 16243, I am waiting for a debugger to attach.
[PYFLYBY]
[PYFLYBY] Details:
[PYFLYBY]   Time             : 2021-03-03 17:48:20 IST
[PYFLYBY]   Host             : testhost
[PYFLYBY]   Process          : 16243
[PYFLYBY]   Username         : raza
[PYFLYBY]   Command line     : /u/raza/reproducer.py
[PYFLYBY]   Exception        : ZeroDivisionError: division by zero
[PYFLYBY]   Traceback        :
[PYFLYBY]       File "/u/raza/reproducer.py", line 10, in <module>
[PYFLYBY]         1/0
[PYFLYBY]       ZeroDivisionError: division by zero
[PYFLYBY] While running /u/raza/.PyCharmCE2019.1/config/scratche, AttributeError in __new__ at /usr/local/python/python3.7/site-packages/typing.py:1005
[PYFLYBY]
[PYFLYBY] Please run:
[PYFLYBY]     ssh -t testhost /usr/local/python/python-3.7/std/bin/py -d 16243
[PYFLYBY]
[PYFLYBY] As process 16243, I am waiting for a debugger to attach.
[PYFLYBY]
[PYFLYBY] Details:
[PYFLYBY]   Time             : 2021-03-03 17:48:32 IST
[PYFLYBY]   Host             : testhost
[PYFLYBY]   Process          : 16243
[PYFLYBY]   Username         : raza
[PYFLYBY]   Command line     : /u/raza/reproducer.py
[PYFLYBY]   Exception        : AttributeError: type object 'Callable' has no attribute '_abc_registry'
[PYFLYBY]   Traceback        :
[PYFLYBY]       File "<string>", line 1, in <module>
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 458, in debugger
[PYFLYBY]         _debug_exception(arg, tty=tty)
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 275, in _debug_exception
[PYFLYBY]         with _DebuggerCtx(tty=tty) as pdb:
[PYFLYBY]       File "/opt/python/python-3.7/lib64/python3.7/contextlib.py", line 112, in __enter__
[PYFLYBY]         return next(self.gen)
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 223, in _DebuggerCtx
[PYFLYBY]         pdb = new_IPdb_instance()
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/pyflyby/_interactive.py", line 1224, in new_IPdb_instance
[PYFLYBY]         app = get_ipython_terminal_app_with_autoimporter()
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/pyflyby/_interactive.py", line 231, in get_ipython_terminal_app_with_autoimporter
[PYFLYBY]         app = _get_or_create_ipython_terminal_app()
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/pyflyby/_interactive.py", line 68, in _get_or_create_ipython_terminal_app
[PYFLYBY]         import IPython
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/IPython/__init__.py", line 56, in <module>
[PYFLYBY]         from .terminal.embed import embed
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/IPython/terminal/embed.py", line 14, in <module>
[PYFLYBY]         from IPython.core.magic import Magics, magics_class, line_magic
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/IPython/core/magic.py", line 20, in <module>
[PYFLYBY]         from . import oinspect
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/IPython/core/oinspect.py", line 26, in <module>
[PYFLYBY]         from typing import Union
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/typing.py", line 1357, in <module>
[PYFLYBY]         class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/typing.py", line 1005, in __new__
[PYFLYBY]         self._abc_registry = extra._abc_registry
[PYFLYBY]       AttributeError: type object 'Callable' has no attribute '_abc_registry'
^CFatal Python error: This thread state must be current when releasing

Current thread 0x00007f584d740740 (most recent call first):
  File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 548 in _sleep_until_debugger_attaches
  File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 580 in wait_for_debugger_to_attach
  File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 439 in debugger
  File "/u/raza/reproducer.py", line 6 in mydebugger
  File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 548 in _sleep_until_debugger_attaches
  File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 580 in wait_for_debugger_to_attach
  File "/usr/local/python/python3.7/site-packages/pyflyby/_dbg.py", line 439 in debugger
  File "/u/raza/reproducer.py", line 6 in mydebugger
[1]    16243 abort (core dumped)  python /u/raza/reproducer.py

[antocuni]

I started to investigate a bit into this and I think it's a mixture of two different problems:

1. calling _debug_exception crashes with AttributeError
2. the process aborts with a Fatal Python error when you try to attach a debugger (or maybe only when you try to attach a debugger but it fails with the AttributeError above, I don't know)

I think that (1) is a result of a wrong installation. Note this part of the traceback:

[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/typing.py", line 1357, in <module>
[PYFLYBY]         class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
[PYFLYBY]       File "/usr/local/python/python3.7/site-packages/typing.py", line 1005, in __new__
[PYFLYBY]         self._abc_registry = extra._abc_registry
[PYFLYBY]       AttributeError: type object 'Callable' has no attribute '_abc_registry'

it is trying to import typing.py from site-packages. However, typing.py is also distributed with the stdlib! I think that by mistake this package was installed and it somehow shadows the builtin one.

I don't know exactly what happened in this specific python installation, but I managed to reproduce the problem by doing the following:

$ conda create -n myenv python=3.7
[...]

$ conda activate myenv
(myenv) $ cd git/deshaw/pyflyby/
(myenv) $ pip install -q -e .
(myenv) qgpu3 pyflyby $ pip install -q typing

# this is the stdlib typing, it works correctly
(myenv) $ python -c 'import typing; print(typing)'
<module 'typing' from '/home/antocuni/.conda/envs/myenv/lib/python3.7/typing.py'>

# force to use the site-package version of typing.py
(myenv) qgpu3 pyflyby $ rm ${CONDA_PREFIX}/lib/python3.7/typing.py
(myenv) qgpu3 pyflyby $ python -c 'import typing; print(typing)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/antocuni/.conda/envs/myenv/lib/python3.7/site-packages/typing.py", line 1359, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/antocuni/.conda/envs/myenv/lib/python3.7/site-packages/typing.py", line 1007, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

As for problem number (2), I didn't manage to reproduce it. I tried:

1. a venv on my laptop using system python (3.8)
2. a conda env with python3.7 on another machine
3. a conda env with python3.8

In all cases, I can successfully attach a debugger to reproducer.py

[sac111gp]

Internal ticket for reference: Prop#279270

[asmeurer]

As discussed with Karl, Arvid, and Shiva, we have removed this from "high priority".