Conda environments not detected

[meatballs]

I've recently changed from miniconda3 and python 3.6 to anaconda3 and python 3.7. As a result, conda environments are not being detected and added as jupyter kernels.

I have nb_conda_kernels 2.1.1 installed in my base environment (python 3.7.0) and ipykernel 5.1.0 in my external environment (also python 3.7.0).

Within my external environment, I can see a kernel.json file in ~\anaconda3\envs\env_name\share\jupyter\kernels\python3 and its content correctly points to the bin\python directory of that environment.

However, jupyter kernelspec list does not show the environment and it's not available in either jupyter notebook or lab.

Update
I noticed that my jupyter_notebook_config.py only had one entry for something else and nothing for nb_conda_kernels. I guessed that means jupyter is using the default KernelSpecManager.

So, I removed that file and restarted jupyter notebook. Kernels all detected and available!!

Even when I put the file back exactly as it was, I can create new environments and they are detected ok.

So - all is now working for me, but is there an issue installing nb_conda_kernels where a jupyter_notebook_config.py file exists?

[mcg1969]

Thanks for the report. nb_conda_kernels doesn't modify jupyter_notebook_config.py; it does, however, modify jupyter_notebook_config.json. I'm really not sure what the interaction is between these two files. I'll take a look at some point when I have the time, but if anyone else can offer a solution/PR, it would be appreciated.

Note that we're up to 2.2.0 on here now. The relevant code is in install.py.

[michaelaye]

I'm not sure this is related, but I recently had a look at my jupyter_notebook_config.py in an editor, without changing anything, at least to my knowledge, and since then, my py37 environment wasn't found anymore, while py36 was still there?
I then went on to reinstall the whole conda py37 environment, but to no avail.
I then tried to conda remove the "nb_conda_kernels" package and received this feedback in the terminal:

$ conda remove nb_conda_kernels                                                                   (py37)
Solving environment: done

## Package Plan ##

  environment location: /Users/klay6683/miniconda3/envs/py37

  removed specs:
    - nb_conda_kernels


The following packages will be REMOVED:

    nb_conda_kernels: 2.2.0-py37_1000 conda-forge

Proceed ([y]/n)?

Preparing transaction: done
Verifying transaction: done
Executing transaction: / + /Users/klay6683/miniconda3/envs/py37/bin/python -m nb_conda_kernels.install --disable
Disabling nb_conda_kernels...
ERROR: The global setting does not match the local setting:
    Global: ENABLED
    Local:  DISABLED
This is typically caused by another configuration file in
the path with a conflicting setting.
Use the --verbose flag for more information.

WARNING conda.core.link:run_script(813): pre-unlink script failed for package conda-forge::nb_conda_kernels-2.2.0-py37_1000
consider notifying the package maintainer
done

I'm not user what's going on there, but I can assure you that I would not deactivate nb_conda_kernels, it's my life blood, and currently I'm quite nervous, in trying for an hour to get back to a working py37 environment. :/

[mcg1969]

@michaelaye a couple of things. First, where exactly is your jupyter_notebook_config.py file located, and what non-trivial content does it have in it? For instance, mine is in ~/.jupyter/; and I can see what's active with this grep command:

grep -v '^\s*\(#.*\)\?$' ~/.jupyter/jupyter_notebook_config.py

where exactly is your jupyter_notebook_config.py located; and what exactly does it have inside it (ignoring the comments)? For instance, I created one in ~/.jupyter,

Second, you could try using the verbose enable/disable commands:

python -m nb_conda_kernels.install -v --enable
python -m nb_conda_kernels.install -v --disable
python -m nb_conda_kernels.install -v --status

This should help us see what's going on.

The error you're seeing above is telling me that, somehow, your jupyter_notebook_config.py file already has some sort of setting in it that is overriding what nb_conda_kernels is trying to achieve locally.

[michaelaye]

I think my interpretation of "not having my py37 env" in the notebook is merely the renaming how it appears as discussed in other GH issues.
But for the records, I executed the --status command from above:

$ python -m nb_conda_kernels.install -v --status                                                                                        (py36)
Determining the status of nb_conda_kernels...
Entry points:
  - conda = nb_conda_kernels.discovery:CondaKernelProvider
  NOTE: entry points not used in Jupyter 5.2.3
jupyter_notebook_config.json entries:
  - /etc/jupyter: <no file>
  - /usr/local/etc/jupyter: <no file>
  * /Users/klay6683/miniconda3/envs/py36/etc/jupyter: {
      "NotebookApp": {
        "nbserver_extensions": {
          "jupyter_nbextensions_configurator": true
        },
        "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
      }
    }
  - /Users/klay6683/.jupyter: {
      "NotebookApp": {
        "nbserver_extensions": {
          "jupyterlab_github": true,
          "jupyter_nbextensions_configurator": true,
          "jupyterlab_git": true,
          "nbresuse": true
        },
        "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
      }
    }
Status: enabled

Is maybe the settings in .jupyter clashing with the settings in the conda env when asking for an uninstall, creating above error message?
Your grep through the jupyter_notebook_config only brings up settings related to my notebook server configuration, like port, ip, certfile, keyfile, password and my GH client_id and _secret.

[robmarkcole]

My envs aren't being found on Mac, but are on Windows. On mac, the contents of jupyter_notebook_config.json are sparse:

{
  "NotebookApp": {
    "nbserver_extensions": {
      "jupyterlab": true
    }
  }
}

The kernels are found however:

Robins-MacBook:~ robincole$ jupyter kernelspec list
Available kernels:
  bluesky          /Users/robincole/Library/Jupyter/kernels/bluesky
  circuitpython    /Users/robincole/Library/Jupyter/kernels/circuitpython
  gophernotes      /Users/robincole/Library/Jupyter/kernels/gophernotes
  javascript       /Users/robincole/Library/Jupyter/kernels/javascript
  micropython      /Users/robincole/Library/Jupyter/kernels/micropython
  nodemcu          /Users/robincole/Library/Jupyter/kernels/nodemcu
  python2          /Users/robincole/Library/Jupyter/kernels/python2
  sat_img_env      /Users/robincole/Library/Jupyter/kernels/sat_img_env
  ubit             /Users/robincole/Library/Jupyter/kernels/ubit
  python3          /Users/robincole/anaconda3/share/jupyter/kernels/python3

UPDATE: Clearly something strange going on with the install:

Robins-MacBook:~ robincole$ conda install nb_conda_kernels
Solving environment: done

# All requested packages already installed.

Robins-MacBook:~ robincole$ python -m nb_conda_kernels.install -v --status
/Users/robincole/anaconda3/bin/python: Error while finding module specification for 'nb_conda_kernels.install' (ModuleNotFoundError: No module named 'nb_conda_kernels')

[mcg1969]

I know this is an old issue but I also don't want to close it in case someone can properly identify a root cause and solution.

Note however that in this comment I am seeing two copies of the nb_conda_kernels configuration. The one in /Users/klay6683/.jupyter should be removed, leaving only the one in sys.prefix where nb_conda_kernels is itself installed.

[FBen3]

Hi, I am experiencing a similar issue on Anaconda3, base(root) environment, python=3.7.3, nb_conda_kernels=2.2.1, ipykernel=5.1.0.

I run:

(base) bash-3.2$ jupyter kernelspec list
Available kernels:
  python3    /Users/benjamin/Library/Jupyter/kernels/python3 

However, when I open Jupyter, there are 2 kernels to chose from, as can be seen here

I also check another folder:

(base) bash-3.2$ cd anaconda3/share/jupyter/kernels/
(base) bash-3.2$ ls
python3

So I don't understand where is Python [conda env:root]. (I was searching for it to modify the names.)
I also don't have a jupyter_notebook_config.py, but a jupyter_notebook_config.json in /Users/benjamin/anaconda3/etc/jupyter/. And it holds:

{
  "NotebookApp": {
    "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
  }
}

[kennethjmyers]

So I was having a similar issue, possibly the same as FBen3, I'm going to try to outline the steps I took to get my kernels showing up the way I want them to:

I wanted to make sure I was starting fresh with kernels so I decided to try this.

Removing /Users/${USER}/Library/Jupyter/kernels fixes the issue.

Well it didn't fix my issues but it seemed to remove the kernels that looked like the ones in FBen3's post.

Next step is to make sure you really follow the steps in this post especially for manually registering a kernel that already exists. I'll try to outline the important ones:

1. Update your conda, idk if this is essential but I haven't been using mine in a while and I wanted to make sure I was on the most up to date.
2. Do conda install nb_conda_kernels on the base environment, I believe this is supposed to auto detect new environments in the future but I actually haven't tested it yet.
3. Do conda activate into the environment you want to add
4. Here are the crucial parts: Make sure ipykernel is installed with conda install notebook ipykernel. I thought nb_conda_kernels should be picking up the new kernel but it was not so I had to do the next step.
5. You need install the kernel. If ipython kernel install doesn't work try ipython kernel install --user, I believe the --user flag is for if your kernelspec is located in /Users/... rather than /usr/.... If you want a to add a specific name do ipython kernel install --user --name [kernel_name] --display-name [kernel_name]. IF YOU DON'T SPECIFY THE NAME, I THINK IT WILL OVERWRITE YOUR PYTHON3 KERNEL. I'm not a 100% sure on that but I think that's why they don't show up.
6. conda deactivate and jupyter kernelspec list. You should hopefully start seeing the env names there. In my case this is

Available kernels:
cs682 /Users/kenn/Library/Jupyter/kernels/cs682
py37 /Users/kenn/Library/Jupyter/kernels/py37
python3 /Users/kenn/Library/Jupyter/kernels/python3

7. jupyter notebook on (base) and you should hopefully see the kernels as normal now
   

[mathematicalmichael]

okay, so I had a bunch of problems with this and it sucked up far too many hours of my life. Google ranked this thread up pretty high, so I'm posting it here for posterity. For anyone trying to do add conda-installed kernels in Docker, here is the process:

FROM jupyter/minimal-notebook:7a0c7325e470
ARG conda_env=py36
COPY environment.yml /home/${NB_USER}/tmp/

RUN cd /home/${NB_USER}/tmp/ && \
    conda env create -p $CONDA_DIR/envs/$conda_env -f environment.yml

# create Python 3.6 environment and link it to jupyter
RUN $CONDA_DIR/envs/${conda_env}/bin/python -m ipykernel install --user --name=${conda_env}
# any additional pip installs:
RUN $CONDA_DIR/envs/${conda_env}/bin/pip install numpy

# prepend to path
ENV PATH $CONDA_DIR/envs/${conda_env}/bin:$PATH
# optional defaulting
ENV CONDA_DEFAULT_ENV ${conda_env}

# some cleanup because we used the docker-stacks images.
RUN fix-permissions $CONDA_DIR && \
    fix-permissions /home/$NB_USER

(--display-name "Other Python" in the ipykernel step if you want a different display name in the Jupyter dropdown, and you can avoid the COPY with conda create changed to conda create --quiet --yes -n $conda_env python=3.6 pip ipython ipykernel )

[hackalog]

Maybe this will offer some insight into what was happening above:

The issue I was trying fix was that on my system (MacOS, python==3.7.6, nb_conda_kernels==2.2.2) my conda environments simply weren't appearing.

In running through some of the troubleshooting above, I noticed that

python -m nb_conda_kernels.install -v --disable

was failing to disable the extension.

$ python -m nb_conda_kernels.install -v --disable
Disabling nb_conda_kernels...
Entry points:
  - conda = nb_conda_kernels.discovery:CondaKernelProvider
Removing from local configuration
Writing config in /Users/hackalog/miniconda3/envs/nlp/etc/jupyter...
jupyter_notebook_config.json entries:
  - /etc/jupyter: <no file>
  - /usr/local/etc/jupyter: <no file>
  * /Users/hackalog/miniconda3/envs/nlp/etc/jupyter: {
      "NotebookApp": {
        "nbserver_extensions": {
          "nb_conda": true
        }
      }
    }
  - /Users/hackalog/.jupyter: <no file>
Status: enabled

Notice the status still enabled at the end.

I traced through that code, and in the end, realized the disabled == True (line 145) code path of https://github.com/Anaconda-Platform/nb_conda_kernels/blob/1b9e821e7cafe388b633ba4fdb8feb3eec815d6c/nb_conda_kernels/install.py#L140-L151
was not getting entered. I can't tell if that's intentional, or just a missed case in the refactored if statement on line 140.

(In my case, status == False, is_enabled_local == False, enable == False, and is_enabled_entry == True, which is why the code was ignored)

When I backed nb_conda_kernels all the way back to 2.1.1 (before that code path was refactored) , I finally succeeded in disabling nb_conda_kernels; i.e. the config file was changed to:

{
  "NotebookApp": {}
}

Upon re-enabling it, I could finally see all my conda environments (yay). The new config file was

{
  "NotebookApp": {
    "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
  }
}

Now, when I attempt to upgrade to nb_conda_kernels 2.2.2, I get the same error as @michaelaye, namely:

Enabling nb_conda_kernels...
ERROR: The global setting does not match the local setting:
    Global: ENABLED
    Local:  DISABLED
This is typically caused by another configuration file in
the path with a conflicting setting.
Use the --verbose flag for more information.

[atg-abhishek]

conda install nb_conda_kernels fixed the problem quite easily since it auto-detects and picks up the existing conda envs

[fcollonval]

@mcg1969 I think this can now be close as there is a way to render environment visible by jupyter tools.

[hackalog]

For future reference (And searchability), what is that way?

…

@mcg1969 <https://github.com/mcg1969> I think this can now be close as there is a way to render environment visible by jupyter tools.

[michaelaye]

if that were true, i guess the whole project should be archived? I'm also unaware of other jupyter tools that scan all conda envs for available kernels.

[fcollonval]

As explain in the README, you can optionally install the virtual conda kernel specs in a place searched for by conda:

1. Create a configuration file for jupyter named jupyter_config.json in the folder returned by jupyter --config-dir.
2. Add the following configuration to install all kernel spec for the current user:

{
  "CondaKernelSpecManager": {
    "kernelspec_path": "--user"
  }
}

3. Execute the command (or open the classical Notebook or JupyterLab UI):

python -m nb_conda_kernels list 

4. Check that the conda environment kernels are discovered by jupyter:

jupyter kernelspec list

The kernelspec path needs to be writable by the users as kernelspecs will be dynamically updated by nb_conda_kernels.

[michaelaye]

@fcollonval I think my jupyter_notebook_config.json is dynamically being created from my jupyter_notebook_config.py in the same folder so I'm not sure one should directly edit it? At least I never created it but still am getting it automatically after setting up new jupyter environments on new machines.

[fcollonval]

The json format and the py format are both acceptable. And configuration are merged from the two files (with JSON having a higher priority). So I'll be surprised if jupyter_notebook_config.json is generated from the Python script. What is true, is that running command like jupyter serverextension enable/disable are editing the JSON file. But at no point should a tool generates it (i.e. removing value from it).

To take a parallel, it is like git config. It modifies your git config file. But nevertheless you can change the config file manually.

There is only one exception for jupyter: if you use the flag --generate-config. But it will ask a confirmation if a file already exists.