jupyter 4 notebooks not launching

[michaelaye]

Installed via conda today.

The dashboard is launching fine. But neither a new notebook or the Ipython 3.2.1 notebooks are launching, they are stuck with this kind of display status:

I checked the error console and the first thing it does not seem to like is a reference to the gist plugin I had installed in IPython 3 but that does not seem everything. I can not afford to stay on this version as I need the notebook to be working, so I'm afraid I can't do any further debugging than this.

I'm guessing a clean install without heritage from IPython 3 would have less problems so any hint how I can do that would be appreciated. Thanks!

[michaelaye]

Oh, I also got this in the terminal, but not much saying there, the line with Creating new notebook in is where it's stuck when trying to launching a new notebook.

$ jupyter notebook .
[I 14:26:36.267 NotebookApp] Serving notebooks from local directory: /Users/klay6683/Dropbox/src/iuvs/notebooks
[I 14:26:36.267 NotebookApp] 0 active kernels
[I 14:26:36.267 NotebookApp] The IPython Notebook is running at: http://localhost:8888/
[I 14:26:36.267 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 14:32:42.581 NotebookApp] Kernel started: 9512e5c4-bb71-4b46-b40b-41d0ec698efe
/Users/klay6683/miniconda3/lib/python3.4/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)
[I 14:33:17.164 NotebookApp] Kernel started: fa794431-84f1-419a-b1cb-51ffaa2ff3e5
/Users/klay6683/miniconda3/lib/python3.4/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)
[I 14:33:32.793 NotebookApp] Creating new notebook in
^C[I 14:35:11.039 NotebookApp] interrupted
Serving notebooks from local directory: /Users/klay6683/Dropbox/src/iuvs/notebooks
2 active kernels
The IPython Notebook is running at: http://localhost:8888/
Shutdown this notebook server (y/[n])? y
[C 14:35:14.455 NotebookApp] Shutdown confirmed
[I 14:35:14.455 NotebookApp] Shutting down kernels
[I 14:35:14.972 NotebookApp] Kernel shutdown: 9512e5c4-bb71-4b46-b40b-41d0ec698efe
[I 14:35:14.972 NotebookApp] Kernel shutdown: fa794431-84f1-419a-b1cb-51ffaa2ff3e5

[michaelaye]

I also can report that new notebooks had been created in the folder where I launched them, and I can open them with IPy3 notebook.

[Carreau]

Remove temporarly your custom.js from ~/.jupyter/custom. the rest of the errors don't matter.

[Carreau]

Also once you remove your custom.js you might want to post it here so that we can fix it.

[michaelaye]

I put the custom.js here as a gist:
https://gist.github.com/9ecb5830ea89c701d142

[michaelaye]

Errm, I deleted the custom.js completely and it's still being found? Still stuck at opening a new notebook.

[Carreau]

So if you are "just" loading extensions (and if gist.js is sufficiently up to date). the see here (TL;DR) , here (longer explanation) on how to write and load extensions in a way which is not subject to race condition.

Which roughly in your case is removing custom.js and execute only once in the notebook:

IPython.notebook.config.update({
  "load_extensions": {"gist":true}
})

[Carreau]

You need to restart the server. and check custom js both in ~/.jupyter and ~.ipython/profile_default

[Carreau]

Try also a force refresh in case Safari is cahing things.

[michaelaye]

After renaming the custom.js in ~/.ipython/profile_default a new notebook now launched.
I cannot execute the above IPython.notebook.config.update() command because IPython does not have a module called notebook anymore?

[Carreau]

Sorry should have precised , the above command should be in the JS console.

If things are found in ~/.ipython/profile_default that you might have a 1/2 IPython 3.x 1/2 IPython 4.x System.

[michaelaye]

That would be the result of the conda install jupyter then. What do I need to remove to make it only IPy 4.x?

[mpacer]

So I just ran into this problem(see below for exact warning output) on the command line as well though my actual notebooks open just fine.

/usr/local/lib/python3.4/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The IPython.kernel package has been deprecated. You should import from ipykernel or jupyter_client instead. "You should import from ipykernel or jupyter_client instead.", ShimWarning)

My guess is that it relates to my previous customization in the ipython profile stuff and though it did copy it over to .jupyter when I first ran the notebook, now the existence of both sets of files is causing some kind of confusion.

[michaelaye]

i have that as well everywhere I run Jupyter/IPy 4.x? Thought that it was just a transitional feature? So I have a messed up set-up?

[mpacer]

Hmm, so maybe you're right and it is a transitional feature…I just moved my ~/.ipython file to ~/.ipythontest and its not producing any errors beyond the warning that we've already pointed out. So it can't just be that…

[minrk]

I don't see this with any of my environments. What do you get from pip list?

[michaelaye]

alabaster (0.7.3)
appnope (0.1.0)
astropy (1.0.4)
autopep8 (1.2)
Babel (1.3)
basemap (1.0.7)
bcolz (0.10.0)
beautifulsoup4 (4.3.2)
binaryornot (0.3.0)
blaze (0.8.2)
bokeh (0.9.1dev-5ce7a15-27-gb689a69)
certifi (14.5.14)
chest (0.2.3)
click (3.3)
cliutils (0.1.3)
clyent (0.4.0)
colorama (0.3.3)
conda (3.16.0)
conda-env (2.4.2)
cookiecutter (1.0.0)
coverage (3.7.1)
coveralls (0.5)
Cython (0.23)
cytoolz (0.7.3)
dask (0.6.0)
datashape (0.4.6)
decorator (4.0.2)
dill (0.2.4)
Diviner (2.4.0b4)
docopt (0.6.2)
docutils (0.12)
flake8 (2.4.1)
flake8-docstrings (0.2.1.post1)
Flask (0.10.1)
GDAL (2.0.0)
graphviz (0.4.5)
greenlet (0.4.7)
h5py (2.5.0)
hapi (1.0, /Users/klay6683/Dropbox/src/hapi)
HeapDict (1.0.0)
html5lib (0.999)
hypothesis (1.10.1)
imageio (1.2)
ipykernel (4.0.3)
ipython (4.0.0)
ipython-genutils (0.1.0)
ipywidgets (4.0.2)
isort (4.0.0)
itsdangerous (0.24)
IUVS (0.3.3, /Users/klay6683/Dropbox/src/iuvs)
jdcal (1.0)
Jinja2 (2.8)
joblib (0.8.4)
jsonschema (2.4.0)
jupyter (1.0.0)
jupyter-client (4.0.0)
jupyter-console (4.0.0)
jupyter-core (4.0.3)
line-profiler (1.0)
llvmlite (0.6.0)
locket (0.2.0)
lxml (3.4.4)
MarkupSafe (0.23)
matplotlib (1.4.3)
mccabe (0.3.1)
memory-profiler (0.33)
mistune (0.7)
moviepy (0.2.2.11)
multipledispatch (0.4.8)
nbconvert (4.0.0)
nbformat (4.0.0)
networkx (1.10)
nose (1.3.7)
notebook (4.0.1)
numba (0.20.0)
numexpr (2.4.3)
numpy (1.9.2)
odo (0.3.0)
openpyxl (2.0.2)
ordereddict (1.1)
p4terrains (0.1, /Users/klay6683/Dropbox/src/p4terrains)
pandas (0.16.2)
partd (0.3.2)
path.py (0.0.0)
pep257 (0.6.0)
pep8 (1.6.2)
pexpect (3.3)
pickleshare (0.5)
Pillow (2.9.0)
pip (7.1.0)
Planet4 (0.4, /Users/klay6683/Dropbox/src/planet4)
planetpy (0.1.0, /Users/klay6683/Dropbox/src/planetpy)
plotly (1.6.13)
pluggy (0.3.0)
ply (3.6)
psutil (2.2.1)
ptyprocess (0.4)
pvl (0.1.1)
py (1.4.30)
PYCISS (0.2, /Users/klay6683/Dropbox/src/pyciss)
pycosat (0.6.1)
pyflakes (0.8.1)
Pygments (2.0.2)
pymongo (2.8)
PyOpenGL (3.1.1a1)
pyparsing (2.0.3)
PyPDS (1.0.1)
Pysis (0.5.2, /Users/klay6683/Dropbox/src/Pysis)
pytest (2.7.2)
pytest-cov (2.0.0)
python-dateutil (2.4.2)
pytz (2015.4)
pyuvis (0.2, /Users/klay6683/Dropbox/src/pyuvis)
PyYAML (3.11)
pyzmq (14.7.0)
qtconsole (4.0.0)
requests (2.7.0)
scikit-image (0.11.3)
scikit-learn (0.16.1)
scipy (0.16.0)
seaborn (0.6.dev0)
setuptools (18.1)
simplegeneric (0.8.1)
simplejson (3.6.5)
six (1.9.0)
snowballstemmer (1.2.0)
Sphinx (1.3.1)
sphinx-rtd-theme (0.1.7)
SpiceyPy (0.5.3)
SQLAlchemy (0.9.9)
tables (3.2.0)
terminado (0.5)
toolz (0.7.4)
tornado (4.2.1)
tox (2.1.1)
tqdm (1.0)
traitlets (0.0.2)
virtualenv (13.1.0)
vispy (0.5.0.dev0, /Users/klay6683/Dropbox/src/vispy)
Werkzeug (0.10.4)
wget (2.2)
wheel (0.24.0)
xlrd (0.9.4)
xonsh (0.1.6, /Users/klay6683/Dropbox/src/xonsh)

[michaelaye]

Oh, and btw, I tried to update with conda today and got an interesting resolution offered:

$ conda update --all
Fetching package metadata: ....
Warning: No packages found in current osx-64 channels matching: xonsh, skipping
Solving package specifications: .
Package plan for installation in environment /Users/klay6683/miniconda3:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    jupyter_core-4.0.4         |           py34_0          26 KB
    notebook-4.0.2             |           py34_0         4.2 MB
    qtconsole-4.0.1            |           py34_0         128 KB
    ------------------------------------------------------------
                                           Total:         4.4 MB

The following packages will be UPDATED:

    jupyter_core: 4.0.3-py34_0 --> 4.0.4-py34_0
    notebook:     4.0.1-py34_0 --> 4.0.2-py34_0
    qtconsole:    4.0.0-py34_0 --> 4.0.1-py34_0

The following packages will be DOWNGRADED:

    ipython:      4.0.0-py34_0 --> 3.2.1-py34_0

Proceed ([y]/n)?

[mpacer]

So @jhamrick just helped me figure out what was going on.

The problem is that the kernelspec has a deprecated command, so while the first time you run jupyter notebook it does migrate your settings, it doesn't upgrade you to non-deprecated versions of the ipython kernel commands. So this issues shouldn't up in non-ipython kernels.

If you want to see the contents of the files where this is specified you can use:
cat /usr/local/share/jupyter/kernels/python2/kernel.json
or
cat ~/Library/Jupyter/kernels/python3/kernel.json

swapping out python2 or python3 depending on where you've installed them (apparently /usr/local/share/jupyter/kernels are where global kernels are stored and ~/Library/Jupyter/kernels/ are where user specific kernels are stored.

For example, my old python2/kernel.json read as follows

{
"display_name": "Python 2",
"language": "python",
"argv": [
    "/usr/local/opt/python/bin/python2.7",
    "-m",
    "IPython.kernel",
    "-f",
    "{connection_file}"
]
}

and my old python3/kernel.json read as:

{
"display_name": "IPython (Python 3)",
"language": "python",
"argv": [
    "python3",
    "-c", "from IPython.kernel.zmq.kernelapp import main; main()",
    "-f", "{connection_file}"
],
"codemirror_mode": {
    "version": 2,
    "name": "ipython"
}
}

To fix it run jupyter kernelspec install-self --user which will fix the incorrect command to ipykernel

So that afterward the Python3 kernel.json reads:

cat ~/Library/Jupyter/kernels/python3/kernel.json

{
"display_name": "Python 3",
"language": "python",
"argv": [
    "/usr/local/opt/python3/bin/python3.4",
    "-m",
    "ipykernel",
    "-f",
    "{connection_file}"
]
}

And

{
"display_name": "Python 2",
"language": "python",
"argv": [
    "/Users/michaelpacer/envs/jupyterpy2/bin/python2.7",
    "-m",
    "ipykernel",
    "-f",
    "{connection_file}"
]
}

Where the python2 path goes to a virtualenv installed version of python since you need to run the jupyter kernelspec install-self --user command from a version of the jupyter that was installed using python2 to update the python2 kernel (or conversely python3 to update the python3 kernel). Because my main python is python3 from which I installed my global version of jupyter, that's why it has a non-virtualenv version of python3 in its path.

Anyway — I dunno if this will fix all your problems, but this should at least make those warnings go away.

[michaelaye]

I executed jupyter kernelspec install-self --user several times (on different installs) successfully to avoid the ShimWarning about kernelspec in the Terminal where I launched jupyter. But now I'm getting it in a ipython notebook running in jupyter inside the cell even after reinstalling the kernel?

%matplotlib notebook
import seaborn as sns
sns.set_context('notebook')

/Users/klay6683/miniconda3/lib/python3.4/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)

For debugging purposes I renamed the old .ipython folder, to no avail.

[minrk]

The warning's probably coming from Seaborn importing IPython.kernel. Nothing wrong, just a warning that a deprecated API is in use. Is anything not working?

[michaelaye]

No, all seems fine. I also get it during an import of one of my own modules, weirdly? Is this hidden in the .pyc files maybe? My libraries are installed via the develop option though, so any change should recreate the pyc.

[michaelaye]

Actually, a fresh compile of MPL removed that warning and now I'm only getting from my module and I certainly don't import kernels unless it's somewhere chained? Would an import of IPython.display or html.widgets create this?

[Carreau]

Actually, a fresh compile of MPL removed that warning and now I'm only getting from my module and I certainly don't import kernels unless it's somewhere chained? Would an import of IPython.display or html.widgets create this?

Probably not,

Try to do the following before anything else:

import warnings
from  IPython.utils.shimmodule import ShimWarning
warnings.filterwarnings('error','', ShimWarning)

This will turn ShimWarnings into errors with tracebacks, so you can narrow down what is raising the warning.

[minrk]

closed by #424

[michaelaye]

Thanks, @Carreau for that tip, I somehow did not see this earlier, sorry.

[ghost]

On one of my computers that I had been using IPython since 2010 early days, I finally upgraded last week to Jupyter and I can confirm that kernels launch fine. Still getting shim warning about IPython.kernel warning and this was the only link that turned up on google search. Based on this thread, sounds like something I should not be too concerned about since everything else seems to be working.