Jupyter running wrong python kernel

[Daniel-M]

Hi everyone, I want to report that Jupyter is loading the wrong Kernel

My machine is a Debian Testing,
Linux 4.9.0-3-amd64 #1 SMP Debian 4.9.30-1 (2017-06-04) x86_64 GNU/Linux

Python2.7 is installed by default, Python3 was installed from repositories, but I'm not sure if it comes by default with the system too. Jupyter and Ipython where installed via Pip (Pip was installed from the repositories)

Module Versions
Installed via pip with dependencies
pip install -U jupyter ipython
pip3 install -U jupyter ipython

So far, the versions installed are,

• Jupyter version 4.3.0 (PIP2 and PIP3)
• Python 3.5.3 from repos
• Python 2.7.13 from repos
• Ipython 5.4.1 (PIP2)
• Ipython3 6.1.0 (PIP3)

Proceeded to install the corresponding kernels as root with the commands,
ipython kernel install
ipython3 kernel install

Which seemed to work since Jupyter recognizes both kernels

And at the location /usr/local/share/jupyter/kernels there's this tree architecture

With the relevant contents,

$ cat python2/kernel.json 
{
 "display_name": "Python 2", 
 "language": "python", 
 "argv": [
  "/usr/bin/python", 
  "-m", 
  "ipykernel_launcher", 
  "-f", 
  "{connection_file}"
 ]
}

$ cat python3/kernel.json 
{
 "display_name": "Python 3",
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "language": "python"
}

I try to program with Python3 standard since 2.7 is old, so Installed a module I wanted to use by means of pip3 install module_name. The module worked with python3 console and Ipython3 console, but when I tried to implement the code inside a Jupyter notebook (with kernel Python3) and I got an error telling that the module wasn't available. After being unable to solve the problem I installed the module with pip2 and suddenly it was available at the Jupyter notebook (with kernel Python3) so I performed a simple check of the version of the running kernel under Jupyter,

As you can see, it says that the kernel is Python3, but the call to sys.version (from the module sys) still shows that the running kernel is Python2.7. I don't know what's going on, even resetted Jupyther's configuration with jupyter notebook --generate-config and the issue remains unsolved, that's why I decided to post here.

On the other hand, calling the deprecated form ipython3 notebook produces the desired result,

Wath am I missing?

[takluyver]

Can you check jupyter kernelspec list to see if there's another kernel spec overriding the one you found?

[Daniel-M]

@takluyver off course, as you thought there was an override of the kernels. The output of jupyter kernelspec list was,

Available kernels:
  python3    /home/damejia/.local/share/jupyter/kernels/python3
  python2    /usr/local/share/jupyter/kernels/python2

The kernel for python3 located in /home/damejia/.local/share/jupyter/kernels/python3 was incorrectly configured. The kernel.json read,

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

So it called the wrong python version (python points to Python 2.7 in my machine). I just removed the /home/damejia/.local/share/jupyter/kernels/ path and now jupyter kernelspec list shows,

Available kernels:
  python2    /usr/local/share/jupyter/kernels/python2
  python3    /usr/local/share/jupyter/kernels/python3

And under a Python3 kernel in Jupyter the output of,

import sys
sys.version

baheves as expected,

'3.5.3 (default, Jan 19 2017, 14:11:04) \n[GCC 6.3.0 20170118]'

I'm not sure why this happened, I've been using Jupyther for about a year without problems so far. I hope this thread can help anyone. I guess It's closed 👍

[takluyver]

@minrk could this be a result of trying to install the kernelspec as a data file in the Python package installation? I notice that it's using the new generic python in the kernel.json.

[Daniel-M]

Well... I'm not sure. how can I test this?

[takluyver]

@Daniel-M no worries, I was asking @minrk if this could be related to a new way kernel specs are being installed. I'm pretty sure it is , having thought about it a bit more.

[minrk]

Yes, this is a result of the data-files installation. One solution is to hardcode handling of 'python2' or 'python3'. Another would be to change the kernelspec name to 'python' and call it Python (default), rather than specify the version (which probably makes more sense to most people, anyway).

[damianavila]

Another would be to change the kernelspec name to 'python' and call it Python (default)

I think is a sensible idea.

[muennemann]

I had the same issue. In addition moving /home/myname/.local/share/jupyter/kernels/, I found that I needed to edit the two kernel.json files under /usr/local/share/jupyter/kernels/. In each of these, I changed "python" to explicitly-named python2 or python3 kernels. That fixed the problem for me.

[Daniel-M]

Quick update

Hi everyone, i'm writing this update since it could help diagnose the problem.
So, after a fresh install of the Debian OS (testing, the weekly build for this week),

Linux ibuki 4.11.0-1-amd64 #1 SMP Debian 4.11.6-1 (2017-06-19) x86_64 GNU/Linux

Module Versions
Installed via pip with dependencies,
pip3 install --upgrade pip jupyter ipython

So far, the versions installed are,

Jupyter version 4.3.0 (via PIP3)
Python 3.5.4rc1 from the Debian repos
Ipython3 6.1.0 (via PIP3)

Once the installation was made via pip3, the kernel for Python3 was automatically installed at
/usr/local/share/jupyter/kernels/python3 and read the wrong python version again as initially reported

Solution

Proceeded to install the corresponding kernel as root with the command,
ipython3 kernel install

[alegarra]

Folks, just for adding a bit of information.

I've been with this kind of problem with Julia notebooks for months. Running Mac OS X. I had installed in succesion Julia 0.4.5, 0.5 and 0.6, from the OSX installers. I wiped 0.4.5 and 0.5 out completely (so I think). Yet I would start a Julia notebook and it would look for Julia 0.4.5 without giving me any choice.

After reading this thread I verified that I had 3 folders still were around:

putterri:anaconda andres$ jupyter kernelspec list
Available kernels:
  ir           /Users/andres/Library/Jupyter/kernels/ir
  julia-0.4    /Users/andres/Library/Jupyter/kernels/julia-0.4
  julia-0.5    /Users/andres/Library/Jupyter/kernels/julia-0.5
  julia-0.6    /Users/andres/Library/Jupyter/kernels/julia-0.6
  python3      /Users/andres/anaconda/share/jupyter/kernels/python3
putterri:anaconda andres$ ls ~/Library/Jupyter/kernels/
ir        julia-0.4 julia-0.5 julia-0.6

It is weird that I have some of these in ~/Library and some in ~/anaconda. In my view this is the Julia installation by package, the anaconda installation by package, and some macports that I have installed (not sure about this one)

Anyway. I deleted these folders

  julia-0.4    /Users/andres/Library/Jupyter/kernels/julia-0.4
  julia-0.5    /Users/andres/Library/Jupyter/kernels/julia-0.5

and things seem to work perfectly now.

[rahulremanan]

I also encountered a similar error. Jupyter notebook installation in my Ubuntu 17.04 cloud machine had issues selecting the right kernel. This is what I did to solve this issue.

Step 1:
Install all the necessary dependencies:
For Python2:
$sudo apt-get install python-setuptools python-dev build-essential
$sudo easy_install pip

$python2 -m pip --version

$python2 -m pip install ipykernel
$sudo python2 -m ipykernel install --user
$pip install -U jupyter ipython

For Python3:
$sudo apt-get install python3-setuptools python3-dev build-essential
$sudo apt-get install python3-pip

$python3 -m pip --version

$sudo python3 -m pip install ipykernel
$sudo python3 -m ipykernel install --user
$sudo pip3 install -U jupyter ipython

Step 2:
Find the python executable locations for each version of Python and available kernels for Jupyter:

For Python2:
$python

import sys
print sys.executable
The output for me was: /usr/bin/python

For Python3:
$python3

import sys
print (sys.executable)
The output for me was: /usr/bin/python3

To get the list of available kernels:

$jupyter kernelspec list

The output for me was:
Available kernels:
python2 /home/info/.local/share/jupyter/kernels/python2
python3 /home/info/.local/share/jupyter/kernels/python3

Step 3:
Update the kernel.json file with the correct location pointers to python executable

For Python2:
$sudo nano /usr/local/share/jupyter/kernels/python2/kernel.json

Eg:

{
"display_name": "Python 2",
"language": "python",
"argv": [
"/usr/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

For Python3:
$sudo nano /usr/local/share/jupyter/kernels/python3/kernel.json

{
"display_name": "Python 3",
"language": "python",
"argv": [
"/usr/bin/python3",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

Step 4:
Restart Jupyter and toggle kernels to see if python versions are selected correctly in the back-end.

[nscozzaro]

Thanks @rahulremanan, as you suggested, I just changed {"argv": ["python",... ]} to {"argv": ["python3",... ]} and it worked. I'm not sure how it got set incorrectly, since it was fine before I upgraded jupyter notebook.

[mgeier]

I'm having a similar problem on binder.
I've used this in postBuild:

python2 -m pip install ipykernel --user
python2 -m ipykernel install --user

I've also tried replacing python2 with /usr/bin/python and python2.7 and /usr/bin/python2.7, but in all cases I get the same broken kernel.json:

{
 "display_name": "Python 2",
 "language": "python",
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ]
}

Strangely, when I run the exact same command in the terminal on the binder instance itself, the kernel.json is overwritten with the correct argv and everything works:

{
 "display_name": "Python 2",
 "language": "python",
 "argv": [
  "/usr/bin/python2",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ]
}

Any idea how to fix this in the binder setup?

[Carreau]

You probably want to ask that on a binder repo, also you should not need to do that on a binder, it should install the kernel automatically.

[mgeier]

I should probably have said that I'm using binder with the (default) Python 3 kernel. And I wanted to install a Python 2 kernel manually on top of that.

[rahulremanan]

@mgeier Can you post the full output for the following:
*** After launching your python 2.x command line interpreter (Eg: $python ) ***

import sys
print sys.executable
print sys.version

[takluyver]

When you do python2 -m ipykernel install --user, does it tell you where it wrote to? If not, try with --debug and see if that does. I wonder if it's writing to the wrong place, so you're not seeing it.

[mgeier]

@takluyver It tells me:

Installed kernelspec python2 in /home/jovyan/.local/share/jupyter/kernels/python2

This is the right directory, but the file doesn't seem to be changed, it's the same generic version (using python) that's installed by pip install ipykernel.
If I run this command directly on the running binder image in the browser-based terminal, it works and kernel.json is changed correctly.

If I add --debug, this happens:

usage: ipython-kernel-install [-h] [--user] [--name NAME]                       
                              [--display-name DISPLAY_NAME]                     
                              [--profile PROFILE] [--prefix PREFIX]             
                              [--sys-prefix]                                    
ipython-kernel-install: error: unrecognized arguments: --debug

[takluyver]

I wonder if there's something special binder does when building images that overrides the kernelspec file you created. @choldgraf @yuvipanda ?

[choldgraf]

hey @mgeier - have you tried using a runtime.txt file? https://mybinder.readthedocs.io/en/latest/preparing.html#using-python2

[mgeier]

@rahulremanan

This is the requested output when running the commands from postBuild:

/usr/bin/python2
2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118]

@choldgraf

No, I haven't tried runtime.txt because that's not what I wanted. I wanted a default vanilla Python 3 binder image and only on top of that a Python 2 kernel.
I didn't try whether installing a Python3 kernel in a Python 2 binder causes the same error.

[deerlord]

$ jupyter kernelspec list --json
{
"kernelspecs": {
"python3": {
"resource_dir": "/home/trent/.local/share/jupyter/kernels/python3",
"spec": {
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env": {},
"display_name": "Python 3",
"language": "python"
}
}
}
}

$ stat /usr/bin/python
File: /usr/bin/python -> /usr/bin/python2

I explicitly installed this with python3, and even Jupyter knows it's python3

$ sudo find / -iname 'kernel.json'
~/.local/share/jupyter/kernels/python3/kernel.json

Yet, it completely ignored the fact that a python3 interpreter was used, and instead tries to use 2.7. Changing argv[0] to the correct path (/usr/bin/python3) in the kernel.json file fixed this (well maybe, I still get an error in Atom/Hydrogen but the output indicates the kernel started).

[choldgraf]

@yuvipanda any ideas on #270 (comment) ?

this feels like a repo2docker bug to me if the postBuild scripts aren't working. Once we figure this out I'll create a repo to demonstrate it in binder-examples

[choldgraf]

opened up binder-examples/python2_with_3#1 to see if we can get a prototype of python 2 within 3 working

[mgeier]

Good news everyone!
It seems to work now, I guess something changed on the binder side.

Just for reference, some more details on what I tried and what worked for me in the end:

I used a requirements.txt, not an environment.yml.

I used an apt.txt with a few Python 2 packages to avoid compiling anything when installing ipykernel:

python-scandir
python-simplegeneric
python-tornado
python-zmq

And I used this postBuild:

#!/bin/sh

set -e

# install pip for Python 2
curl -O https://bootstrap.pypa.io/get-pip.py
python2 get-pip.py --user

# install and register Python 2 kernel
python2 -m pip install ipykernel --user
python2 -m ipykernel install --user

[choldgraf]

Just FYI that there's now a working solution in https://github.com/binder-examples/python2_with_3 that uses runtime.txt with requirements3.txt to install python 3 requirements as needed

[blink1073]

I think the most recent problems are due to me accidentally publishing a universal wheel for ipykernel on December 6th. cf ipython/ipykernel#288 (comment)

[ibell]

This is still an open and annoying problem - I have two python 3.6 environments set up for different projects (that is, after all, the whole point of having conda environments in the first place). It seems that the kernelspec will only allow one of them to be the python 3.6 kernel. That seems like an architectural mistake - each environment should know about the kernels associated with it. Or am I mistaken? Is there a way to handle multiple python versions aside from manually modifying the kernel file each time (ugh...)?

[takluyver]

If you want to use kernels from conda environments, you can install a package called something like nb_conda_kernels which provides kernels based on conda environments.

[takluyver]

Or you can install a different kernelspec file for each environment (i.e. use names other than python3).

[dashesy]

this looks like a similar issue but on Windows

[GraemeHerrington]

Same issue - changing the default on Mint to 3.6 from 2.7 broke Jupyter as described.

@rahulremanan's solution worked for me - editing the kernel.json file to point directly at python 2.7 and python 3.6. Checked separate kernels in the same run copy of Jupyter and the outputs are now in line with expectation.