Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jupyter running wrong python kernel #270

Open
Daniel-M opened this issue Jun 9, 2017 · 43 comments
Open

Jupyter running wrong python kernel #270

Daniel-M opened this issue Jun 9, 2017 · 43 comments

Comments

@Daniel-M
Copy link

@Daniel-M Daniel-M commented Jun 9, 2017

Hi everyone, I want to report that Jupyter is loading the wrong Kernel

My machine is a Debian Testing,
Linux 4.9.0-3-amd64 #1 SMP Debian 4.9.30-1 (2017-06-04) x86_64 GNU/Linux

Python2.7 is installed by default, Python3 was installed from repositories, but I'm not sure if it comes by default with the system too. Jupyter and Ipython where installed via Pip (Pip was installed from the repositories)

Module Versions
Installed via pip with dependencies
pip install -U jupyter ipython
pip3 install -U jupyter ipython

So far, the versions installed are,

  • Jupyter version 4.3.0 (PIP2 and PIP3)
  • Python 3.5.3 from repos
  • Python 2.7.13 from repos
  • Ipython 5.4.1 (PIP2)
  • Ipython3 6.1.0 (PIP3)

Proceeded to install the corresponding kernels as root with the commands,
ipython kernel install
ipython3 kernel install

Which seemed to work since Jupyter recognizes both kernels

image

And at the location /usr/local/share/jupyter/kernels there's this tree architecture

image

With the relevant contents,

$ cat python2/kernel.json 
{
 "display_name": "Python 2", 
 "language": "python", 
 "argv": [
  "/usr/bin/python", 
  "-m", 
  "ipykernel_launcher", 
  "-f", 
  "{connection_file}"
 ]
}
$ cat python3/kernel.json 
{
 "display_name": "Python 3",
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "language": "python"
}

I try to program with Python3 standard since 2.7 is old, so Installed a module I wanted to use by means of pip3 install module_name. The module worked with python3 console and Ipython3 console, but when I tried to implement the code inside a Jupyter notebook (with kernel Python3) and I got an error telling that the module wasn't available. After being unable to solve the problem I installed the module with pip2 and suddenly it was available at the Jupyter notebook (with kernel Python3) so I performed a simple check of the version of the running kernel under Jupyter,

image

As you can see, it says that the kernel is Python3, but the call to sys.version (from the module sys) still shows that the running kernel is Python2.7. I don't know what's going on, even resetted Jupyther's configuration with jupyter notebook --generate-config and the issue remains unsolved, that's why I decided to post here.

On the other hand, calling the deprecated form ipython3 notebook produces the desired result,

image

Wath am I missing?

@Daniel-M Daniel-M changed the title Jupyter running wrong kernel Jupyter running wrong python kernel Jun 9, 2017
@takluyver
Copy link
Member

@takluyver takluyver commented Jun 9, 2017

Can you check jupyter kernelspec list to see if there's another kernel spec overriding the one you found?

@Daniel-M
Copy link
Author

@Daniel-M Daniel-M commented Jun 9, 2017

@takluyver off course, as you thought there was an override of the kernels. The output of jupyter kernelspec list was,

Available kernels:
  python3    /home/damejia/.local/share/jupyter/kernels/python3
  python2    /usr/local/share/jupyter/kernels/python2

The kernel for python3 located in /home/damejia/.local/share/jupyter/kernels/python3 was incorrectly configured. The kernel.json read,

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

So it called the wrong python version (python points to Python 2.7 in my machine). I just removed the /home/damejia/.local/share/jupyter/kernels/ path and now jupyter kernelspec list shows,

Available kernels:
  python2    /usr/local/share/jupyter/kernels/python2
  python3    /usr/local/share/jupyter/kernels/python3

And under a Python3 kernel in Jupyter the output of,

import sys
sys.version

baheves as expected,

'3.5.3 (default, Jan 19 2017, 14:11:04) \n[GCC 6.3.0 20170118]'

I'm not sure why this happened, I've been using Jupyther for about a year without problems so far. I hope this thread can help anyone. I guess It's closed 👍

@Daniel-M Daniel-M closed this Jun 9, 2017
@takluyver takluyver reopened this Jun 9, 2017
@takluyver
Copy link
Member

@takluyver takluyver commented Jun 9, 2017

@minrk could this be a result of trying to install the kernelspec as a data file in the Python package installation? I notice that it's using the new generic python in the kernel.json.

@Daniel-M
Copy link
Author

@Daniel-M Daniel-M commented Jun 9, 2017

Well... I'm not sure. how can I test this?

@takluyver
Copy link
Member

@takluyver takluyver commented Jun 12, 2017

@Daniel-M no worries, I was asking @minrk if this could be related to a new way kernel specs are being installed. I'm pretty sure it is , having thought about it a bit more.

@minrk
Copy link
Member

@minrk minrk commented Jun 12, 2017

Yes, this is a result of the data-files installation. One solution is to hardcode handling of 'python2' or 'python3'. Another would be to change the kernelspec name to 'python' and call it Python (default), rather than specify the version (which probably makes more sense to most people, anyway).

@damianavila
Copy link
Member

@damianavila damianavila commented Jun 13, 2017

Another would be to change the kernelspec name to 'python' and call it Python (default)

I think is a sensible idea.

@muennemann
Copy link

@muennemann muennemann commented Jul 25, 2017

I had the same issue. In addition moving /home/myname/.local/share/jupyter/kernels/, I found that I needed to edit the two kernel.json files under /usr/local/share/jupyter/kernels/. In each of these, I changed "python" to explicitly-named python2 or python3 kernels. That fixed the problem for me.

@Daniel-M
Copy link
Author

@Daniel-M Daniel-M commented Aug 17, 2017

Quick update

Hi everyone, i'm writing this update since it could help diagnose the problem.
So, after a fresh install of the Debian OS (testing, the weekly build for this week),

Linux ibuki 4.11.0-1-amd64 #1 SMP Debian 4.11.6-1 (2017-06-19) x86_64 GNU/Linux

Module Versions
Installed via pip with dependencies,
pip3 install --upgrade pip jupyter ipython

So far, the versions installed are,

Jupyter version 4.3.0 (via PIP3)
Python 3.5.4rc1 from the Debian repos
Ipython3 6.1.0 (via PIP3)

Once the installation was made via pip3, the kernel for Python3 was automatically installed at
/usr/local/share/jupyter/kernels/python3 and read the wrong python version again as initially reported

Solution

Proceeded to install the corresponding kernel as root with the command,
ipython3 kernel install

@alegarra
Copy link

@alegarra alegarra commented Sep 13, 2017

Folks, just for adding a bit of information.

I've been with this kind of problem with Julia notebooks for months. Running Mac OS X. I had installed in succesion Julia 0.4.5, 0.5 and 0.6, from the OSX installers. I wiped 0.4.5 and 0.5 out completely (so I think). Yet I would start a Julia notebook and it would look for Julia 0.4.5 without giving me any choice.

After reading this thread I verified that I had 3 folders still were around:

putterri:anaconda andres$ jupyter kernelspec list
Available kernels:
  ir           /Users/andres/Library/Jupyter/kernels/ir
  julia-0.4    /Users/andres/Library/Jupyter/kernels/julia-0.4
  julia-0.5    /Users/andres/Library/Jupyter/kernels/julia-0.5
  julia-0.6    /Users/andres/Library/Jupyter/kernels/julia-0.6
  python3      /Users/andres/anaconda/share/jupyter/kernels/python3
putterri:anaconda andres$ ls ~/Library/Jupyter/kernels/
ir        julia-0.4 julia-0.5 julia-0.6

It is weird that I have some of these in ~/Library and some in ~/anaconda. In my view this is the Julia installation by package, the anaconda installation by package, and some macports that I have installed (not sure about this one)

Anyway. I deleted these folders

  julia-0.4    /Users/andres/Library/Jupyter/kernels/julia-0.4
  julia-0.5    /Users/andres/Library/Jupyter/kernels/julia-0.5

and things seem to work perfectly now.

@rahulremanan
Copy link

@rahulremanan rahulremanan commented Dec 2, 2017

I also encountered a similar error. Jupyter notebook installation in my Ubuntu 17.04 cloud machine had issues selecting the right kernel. This is what I did to solve this issue.

Step 1:
Install all the necessary dependencies:
For Python2:
$sudo apt-get install python-setuptools python-dev build-essential
$sudo easy_install pip

$python2 -m pip --version

$python2 -m pip install ipykernel
$sudo python2 -m ipykernel install --user
$pip install -U jupyter ipython

For Python3:
$sudo apt-get install python3-setuptools python3-dev build-essential
$sudo apt-get install python3-pip

$python3 -m pip --version

$sudo python3 -m pip install ipykernel
$sudo python3 -m ipykernel install --user
$sudo pip3 install -U jupyter ipython

Step 2:
Find the python executable locations for each version of Python and available kernels for Jupyter:

For Python2:
$python

import sys
print sys.executable
The output for me was: /usr/bin/python

For Python3:
$python3

import sys
print (sys.executable)
The output for me was: /usr/bin/python3

To get the list of available kernels:

$jupyter kernelspec list

The output for me was:
Available kernels:
python2 /home/info/.local/share/jupyter/kernels/python2
python3 /home/info/.local/share/jupyter/kernels/python3

Step 3:
Update the kernel.json file with the correct location pointers to python executable

For Python2:
$sudo nano /usr/local/share/jupyter/kernels/python2/kernel.json

Eg:

{
"display_name": "Python 2",
"language": "python",
"argv": [
"/usr/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

For Python3:
$sudo nano /usr/local/share/jupyter/kernels/python3/kernel.json

{
"display_name": "Python 3",
"language": "python",
"argv": [
"/usr/bin/python3",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

Step 4:
Restart Jupyter and toggle kernels to see if python versions are selected correctly in the back-end.

@nscozzaro
Copy link

@nscozzaro nscozzaro commented Dec 3, 2017

Thanks @rahulremanan, as you suggested, I just changed {"argv": ["python",... ]} to {"argv": ["python3",... ]} and it worked. I'm not sure how it got set incorrectly, since it was fine before I upgraded jupyter notebook.

@mgeier
Copy link

@mgeier mgeier commented Dec 5, 2017

I'm having a similar problem on binder.
I've used this in postBuild:

python2 -m pip install ipykernel --user
python2 -m ipykernel install --user

I've also tried replacing python2 with /usr/bin/python and python2.7 and /usr/bin/python2.7, but in all cases I get the same broken kernel.json:

{
 "display_name": "Python 2",
 "language": "python",
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ]
}

Strangely, when I run the exact same command in the terminal on the binder instance itself, the kernel.json is overwritten with the correct argv and everything works:

{
 "display_name": "Python 2",
 "language": "python",
 "argv": [
  "/usr/bin/python2",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ]
}

Any idea how to fix this in the binder setup?

@Carreau
Copy link
Member

@Carreau Carreau commented Dec 5, 2017

You probably want to ask that on a binder repo, also you should not need to do that on a binder, it should install the kernel automatically.

@mgeier
Copy link

@mgeier mgeier commented Dec 5, 2017

I should probably have said that I'm using binder with the (default) Python 3 kernel. And I wanted to install a Python 2 kernel manually on top of that.

@rahulremanan
Copy link

@rahulremanan rahulremanan commented Dec 5, 2017

@mgeier Can you post the full output for the following:
*** After launching your python 2.x command line interpreter (Eg: $python ) ***

import sys
print sys.executable
print sys.version

@takluyver
Copy link
Member

@takluyver takluyver commented Dec 5, 2017

When you do python2 -m ipykernel install --user, does it tell you where it wrote to? If not, try with --debug and see if that does. I wonder if it's writing to the wrong place, so you're not seeing it.

@mgeier
Copy link

@mgeier mgeier commented Dec 7, 2017

@takluyver It tells me:

Installed kernelspec python2 in /home/jovyan/.local/share/jupyter/kernels/python2

This is the right directory, but the file doesn't seem to be changed, it's the same generic version (using python) that's installed by pip install ipykernel.
If I run this command directly on the running binder image in the browser-based terminal, it works and kernel.json is changed correctly.

If I add --debug, this happens:

usage: ipython-kernel-install [-h] [--user] [--name NAME]                       
                              [--display-name DISPLAY_NAME]                     
                              [--profile PROFILE] [--prefix PREFIX]             
                              [--sys-prefix]                                    
ipython-kernel-install: error: unrecognized arguments: --debug
@takluyver
Copy link
Member

@takluyver takluyver commented Dec 7, 2017

I wonder if there's something special binder does when building images that overrides the kernelspec file you created. @choldgraf @yuvipanda ?

@choldgraf
Copy link
Collaborator

@choldgraf choldgraf commented Dec 7, 2017

@mgeier
Copy link

@mgeier mgeier commented Dec 8, 2017

@rahulremanan

This is the requested output when running the commands from postBuild:

/usr/bin/python2
2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118]

@choldgraf

No, I haven't tried runtime.txt because that's not what I wanted. I wanted a default vanilla Python 3 binder image and only on top of that a Python 2 kernel.
I didn't try whether installing a Python3 kernel in a Python 2 binder causes the same error.

@deerlord
Copy link

@deerlord deerlord commented Dec 8, 2017

$ jupyter kernelspec list --json
{
"kernelspecs": {
"python3": {
"resource_dir": "/home/trent/.local/share/jupyter/kernels/python3",
"spec": {
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env": {},
"display_name": "Python 3",
"language": "python"
}
}
}
}

$ stat /usr/bin/python
File: /usr/bin/python -> /usr/bin/python2

I explicitly installed this with python3, and even Jupyter knows it's python3

$ sudo find / -iname 'kernel.json'
~/.local/share/jupyter/kernels/python3/kernel.json

Yet, it completely ignored the fact that a python3 interpreter was used, and instead tries to use 2.7. Changing argv[0] to the correct path (/usr/bin/python3) in the kernel.json file fixed this (well maybe, I still get an error in Atom/Hydrogen but the output indicates the kernel started).

@choldgraf
Copy link
Collaborator

@choldgraf choldgraf commented Dec 9, 2017

@yuvipanda any ideas on #270 (comment) ?

this feels like a repo2docker bug to me if the postBuild scripts aren't working. Once we figure this out I'll create a repo to demonstrate it in binder-examples

@choldgraf
Copy link
Collaborator

@choldgraf choldgraf commented Dec 9, 2017

opened up binder-examples/python2_with_3#1 to see if we can get a prototype of python 2 within 3 working

@mgeier
Copy link

@mgeier mgeier commented Dec 9, 2017

Good news everyone!
It seems to work now, I guess something changed on the binder side.

Just for reference, some more details on what I tried and what worked for me in the end:

I used a requirements.txt, not an environment.yml.

I used an apt.txt with a few Python 2 packages to avoid compiling anything when installing ipykernel:

python-scandir
python-simplegeneric
python-tornado
python-zmq

And I used this postBuild:

#!/bin/sh

set -e

# install pip for Python 2
curl -O https://bootstrap.pypa.io/get-pip.py
python2 get-pip.py --user

# install and register Python 2 kernel
python2 -m pip install ipykernel --user
python2 -m ipykernel install --user
@choldgraf
Copy link
Collaborator

@choldgraf choldgraf commented Dec 13, 2017

Just FYI that there's now a working solution in https://github.com/binder-examples/python2_with_3 that uses runtime.txt with requirements3.txt to install python 3 requirements as needed

@blink1073
Copy link
Member

@blink1073 blink1073 commented Dec 13, 2017

I think the most recent problems are due to me accidentally publishing a universal wheel for ipykernel on December 6th. cf ipython/ipykernel#288 (comment)

@ibell
Copy link

@ibell ibell commented Jan 6, 2018

This is still an open and annoying problem - I have two python 3.6 environments set up for different projects (that is, after all, the whole point of having conda environments in the first place). It seems that the kernelspec will only allow one of them to be the python 3.6 kernel. That seems like an architectural mistake - each environment should know about the kernels associated with it. Or am I mistaken? Is there a way to handle multiple python versions aside from manually modifying the kernel file each time (ugh...)?

@takluyver
Copy link
Member

@takluyver takluyver commented Jan 8, 2018

If you want to use kernels from conda environments, you can install a package called something like nb_conda_kernels which provides kernels based on conda environments.

@takluyver
Copy link
Member

@takluyver takluyver commented Jan 8, 2018

Or you can install a different kernelspec file for each environment (i.e. use names other than python3).

@dashesy
Copy link

@dashesy dashesy commented May 5, 2018

this looks like a similar issue but on Windows

@GraemeHerrington
Copy link

@GraemeHerrington GraemeHerrington commented Oct 30, 2018

Same issue - changing the default on Mint to 3.6 from 2.7 broke Jupyter as described.

@rahulremanan's solution worked for me - editing the kernel.json file to point directly at python 2.7 and python 3.6. Checked separate kernels in the same run copy of Jupyter and the outputs are now in line with expectation.

@ghost
Copy link

@ghost ghost commented Mar 16, 2019

Quick update

Hi everyone, i'm writing this update since it could help diagnose the problem.
So, after a fresh install of the Debian OS (testing, the weekly build for this week),

Linux ibuki 4.11.0-1-amd64 #1 SMP Debian 4.11.6-1 (2017-06-19) x86_64 GNU/Linux

Module Versions
Installed via pip with dependencies,
pip3 install --upgrade pip jupyter ipython

So far, the versions installed are,

Jupyter version 4.3.0 (via PIP3)
Python 3.5.4rc1 from the Debian repos
Ipython3 6.1.0 (via PIP3)

Once the installation was made via pip3, the kernel for Python3 was automatically installed at
/usr/local/share/jupyter/kernels/python3 and read the wrong python version again as initially reported

Solution

Proceeded to install the corresponding kernel as root with the command,
ipython3 kernel install

Actually this solution helped me. I followed exactly and got resolved.

@stevejryan
Copy link

@stevejryan stevejryan commented Mar 23, 2019

Just ran into what I think is the same problem on Windows.

  • Base Anaconda is python 2.7
  • I have an environment installed which is python 3.7.2
  • This environment folder in conda has its own kernel.json file that points to the correct python executable, in its own environment
  • I have installed nb_conda_kernels just in case that helps

When I run jupyter lab from the conda terminal inside my python 3 environment, it still uses the kernel from another environment (not the base) and doesn't recognize any other kernels as valid.

@ClausTrost
Copy link

@ClausTrost ClausTrost commented Apr 26, 2019

Just ran into what I think is the same problem on Windows.

* Base Anaconda is python 2.7

* I have an environment installed which is python 3.7.2

* This environment folder in conda has its own kernel.json file that points to the correct python executable, in its own environment

* I have installed nb_conda_kernels just in case that helps

When I run jupyter lab from the conda terminal inside my python 3 environment, it still uses the kernel from another environment (not the base) and doesn't recognize any other kernels as valid.

Maybe try the AnacondaNavigator go to "Environments" and open a Jupyter Notebook by klicking on the "Run Button" (Green Recktangle) of your desired Python Version. That worked just fine for me.

@professorlust
Copy link

@professorlust professorlust commented Jul 2, 2019

I also just ran into this error on the new(ish) Google Cloud AI Platform Notebooks.

In both the Tensorflow 1.x and the Python quickstart frameworks, despite having the Python 3 Kernel selected, an error was thrown because the notebook was using python 2.7 instead.

Reporting this here just for awareness that this issue is affecting that platform as well.

@inspiringsource
Copy link

@inspiringsource inspiringsource commented Sep 25, 2019

I also just ran into this error on the new(ish) Google Cloud AI Platform Notebooks.

In both the Tensorflow 1.x and the Python quickstart frameworks, despite having the Python 3 Kernel selected, an error was thrown because the notebook was using python 2.7 instead.

Reporting this here just for awareness that this issue is affecting that platform as well.

I have the same issue...

@danlyth
Copy link

@danlyth danlyth commented Oct 9, 2019

I also just ran into this error on the new(ish) Google Cloud AI Platform Notebooks.
In both the Tensorflow 1.x and the Python quickstart frameworks, despite having the Python 3 Kernel selected, an error was thrown because the notebook was using python 2.7 instead.
Reporting this here just for awareness that this issue is affecting that platform as well.

I have the same issue...

Me too...

@abrunner73
Copy link

@abrunner73 abrunner73 commented Nov 21, 2019

Can you check jupyter kernelspec list to see if there's another kernel spec overriding the one you found?

Thank you @takluyver . Using his suggestion and also sys.executable showed that my kernel was pointing to a python.exe in the wrong virtual environment. Once I updated the kernel.json for the environment I wished to use to point it to the python.exe in the correct environment my issue was resolved. That and ensuring that ipykernel libary was installed in the env. I got myself into this situation because I am still learning Jupyter Lab and how it is set up.

@jelmd
Copy link

@jelmd jelmd commented Nov 29, 2019

Same thing in a Ubuntu 18.04 (bionic) container. Executed as unprivileged user python3 -m ipykernel install produces ~/.local/share/jupyter/kernels/python3/kernel.json:

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

Executing the same command as root produces /usr/local/share/jupyter/kernels/python3/kernel.json:

{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}
@niamrox
Copy link

@niamrox niamrox commented Apr 13, 2020

Quick update
Hi everyone, i'm writing this update since it could help diagnose the problem.
So, after a fresh install of the Debian OS (testing, the weekly build for this week),

Linux ibuki 4.11.0-1-amd64 #1 SMP Debian 4.11.6-1 (2017-06-19) x86_64 GNU/Linux

Module Versions
Installed via pip with dependencies,
pip3 install --upgrade pip jupyter ipython
So far, the versions installed are,
Jupyter version 4.3.0 (via PIP3)
Python 3.5.4rc1 from the Debian repos
Ipython3 6.1.0 (via PIP3)
Once the installation was made via pip3, the kernel for Python3 was automatically installed at
/usr/local/share/jupyter/kernels/python3 and read the wrong python version again as initially reported
Solution
Proceeded to install the corresponding kernel as root with the command,
ipython3 kernel install

Actually this solution helped me. I followed exactly and got resolved.

It Works

@tashrifbillah
Copy link

@tashrifbillah tashrifbillah commented Jun 29, 2020

Does anyone know how to start jupyter-notebook with the name of a kernel or path of a json file in first place?

To be more specific, I am looking for one of the following:

jupyter-notebook my_kernel
jupyter-notebook /path/to/my_kernel/kernel.json
@Anirudh257
Copy link

@Anirudh257 Anirudh257 commented Aug 17, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet