Seqfault in on_trait_change

[jenshnielsen]

With the release of Traitlets 4.1 I am seeing some random seqfaults when restarting the Jupyter notebook and as part of the Matplotlib docs build which seems to point to the Traitlets tear down steps.

Im not sure this is related to Traitlets 4.1 but it does seem to correlate. Perhaps we are doing something wrong in Matplotlib which is causing this.

Locally:

Fatal Python error: PyImport_GetModuleDict: no module dictionary!

Thread 0x0000700001113000 (most recent call first):
  File "/usr/local/lib/python3.5/site-packages/ipykernel/parentpoller.py", line 37 in run
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x0000700000607000 (most recent call first):
  File "/usr/local/lib/python3.5/site-packages/ipykernel/heartbeat.py", line 61 in run
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 882 in _bootstrap

Current thread 0x00007fff7844b000 (most recent call first):
  File "/usr/local/lib/python3.5/site-packages/traitlets/traitlets.py", line 1136 in on_trait_change
  File "/usr/local/lib/python3.5/site-packages/IPython/core/extensions.py", line 64 in __del__

and in the Matplotlib docs build which uses the IPython Sphinx extension:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python3.5/site-packages/IPython/core/history.py", line 547, in end_session
    len(self.input_hist_parsed)-1, self.session_number))
sqlite3.OperationalError: disk I/O error
Fatal Python error: PyImport_GetModuleDict: no module dictionary!
Current thread 0x00007ff58684b700 (most recent call first):
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python3.5/site-packages/traitlets/traitlets.py", line 1136 in on_trait_change
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python3.5/site-packages/IPython/core/extensions.py", line 64 in __del__
Aborted (core dumped)

The sql error is old and fixed already in IPython master see ipython/ipython#8850

[jenshnielsen]

BTW the Matplotlib build log is in https://travis-ci.org/matplotlib/matplotlib/jobs/103175772

[jenshnielsen]

Looking over Matplotlib log we only seem to see it with python 3 (Specifically python 3.5) On the branches where the docs are build with python 2.7 this doesn't happen.

[jenshnielsen]

Im also seeing it in a fresh conda environment

In [1]:
Do you really want to exit ([y]/n)? y
Fatal Python error: PyImport_GetModuleDict: no module dictionary!

Current thread 0x00007fff7844b000 (most recent call first):
  File "/Users/jhn/miniconda3/lib/python3.5/site-packages/traitlets/traitlets.py", line 1136 in on_trait_change
  File "/Users/jhn/miniconda3/lib/python3.5/site-packages/IPython/core/extensions.py", line 64 in __del__
/Users/jhn/miniconda3/bin/python.app: line 3:  4033 Abort trap: 6           /Users/jhn/miniconda3/python.app/Contents/MacOS/python "$@"

[minrk]

I can see this, too sometimes. Seems like it has to be a bug in Python 3.5, even if it's triggered by a change in traitlets. I'll try to see if I can figure out what caused it.

[jenshnielsen]

@minrk Thanks, It sounds plausible that it's a bug in Python 3.5

[minrk]

I just need to narrow it down to what's the exact cause. I've seen it, but I cannot reproduce it reliably, which makes it harder to find what is the relevant change. I can only reproduce the error about 1% of the time, which suggests to me that it might be threading related.

[minrk]

I think I found it: the cause appears to be the deprecation warning that we throw, since on_trait_change is deprecated. I'm guessing it's the inspection used to show the warning message that's failing.

Unfortunately, I don't know a way to fix this in traitlets, since it seems like the only thing we are doing is a simple warning.

Fortunately, though, the __del__ call here that's the direct cause is unnecessary and already removed in master (ipython/ipython#9040). My proposal is to backport this fix and immediately release IPython 4.0.3, while we work with CPython to figure out the upstream bug.

[jenshnielsen]

@minrk Sounds good to me

[minrk]

Released IPython 4.0.3 with the patch. Hopefully that should tide us over while we work out what's up with Python 3.5.

[jenshnielsen]

@minrk Thanks, I will upgrade and see what happens.