What does nb_conda_kernels do?

[ianozsvald]

nb_conda_kernels got added to my Anaconda installation on an update a few months back. Since then I've been confused as to what it does - it seems to make sharing Notebooks harder. What am I missing?

Here's my workflow:

• Create a new environment for a project using conda
• Make some Notebooks, commit to git, share to collaborator
• Collaborator makes a similar environment, clones my repo
• Using an activated environment the collaborator, having the same named project environment, is asked to select an environment (and presumably this changes the metadata for the Notebook)

I don't understand why this just doesn't use the default environment that jupyter notebook was run from. What's the advantage to me of activating an environment, starting a Notebook, then having to choose the same environment (and/or having a collaborator mess with the metadata when we're sharing Notebooks)?

What's the thinking behind fixing a Notebook to a (edit - removed "machine") specific environment name?

Update - I've now upgraded to 2.0 (which still required me to choose a kernel when I started a Notebook), I see on the installation page that I could also run
python -m nb_conda_kernels.install --enable --prefix="%CONDA_PREFIX"
which I've done because it looks magic, but not because I know what it'll fix.

For reference the Notebook metadata in this case contains:

  "kernelspec": {
    "name": "conda-env-fds-py",
    "display_name": "Python [conda env:fds]",
    "language": "python"
  },```

Hoping someone can set me straight...

[ianozsvald]

After some more hacking my colleague tried python -m nb_conda_kernels.install --disable --prefix=$CONDA_PREFIX and this seems to disable the kernels. If we uninstalled the nb_conda packages then they just seemed to get reinstalled after a conda update --all so that didn't seem like a useful solution.

Having run the --disable it seems that when I run my Jupyter Notebook from an activated environment, it uses that environment only and the Notebook Metadata doesn't say anything about a conda-specific environment. I'm hoping this means my colleague and I can share Notebooks without any machine-specific environment naming issues.

There's also a note here https://docs.continuum.io/anaconda/jupyter-notebook-extensions on Uninstalling the extensions and disabling them.

If there's a really good reason not to disable the nb_conda_kernels I'd be happy to be educated. Cheers, Ian.

[jankatins]

[note: most of this information is based on my usage of the environment_kernel_manager which more or less has the same idea as nb_conda_kernels]

The kernel isn't specific to the machine, it uses a name defined by the conda environment name: in this case the nb_conda_kernels creates a (virtual) kernel spec for each conda environment (which has a kernel installed?). So if you both have nb_conda_kernels installed and the same name for the projects conda environment, it should work.

Just to comment on the idea for a newish workflow:

• install jupyter notebook in a new/root environment which only contains the notebook (i.e. no "project related" stuff) -> you only need to install the notebook once.
• create a new environment for your project via conda (e.g. via a environment.yml file in your project dir) and install IPykernel in there
• start the notebook server from the root/first environment in the project directory (e.g. via full path to jupyter.exe or whatever it is called on mac/linux)
• create a new notebook by choosing the kernel from the project environment -> you don't need to activate the env!
  [... hack and share ...]
• get your collaborator to create the same environment(s), especially an environment with the same name as your project environment
• open notebook and it should work

The "default name" is actually depended on having a jupyter kernel in the same environment as the jupyter notebook, which does not need to be the case (at least I tried to remove it, but conda didn't let me, bu AFAIK there shouldn't be a technical reason other than not confusing the user).

[jankatins]

BTW: IMO the notebook (or a new package) should become a meta package which installs all the package which the notebook now depends on in addition to a new "only the notebook" package. That way you could remove the meta package and everything you don't like (or need, so my usecase of not installing a kernel next to the notebook server).

[ianozsvald]

Maybe my confusion came from having older Notebooks (prior to nb_conda_kernels being added to anaconda), then being forced to choose kernels in each and every one, then having my colleague do the same, then having to seemingly go round in circles causing a bunch of frustration. This was prior to version 2.0 which maybe fixes this.

I agree that the kernal names are machine inspecific (I've edited the original post), it felt like it might be specific as with kernel name "fds" set in the metadata by me and my colleague with an environment called "fds" he still had to manually choose his "fds" environment (and conda had already been used to activate his "fds" environment). I'm putting that down to random weirdness as it doesn't seem to happen now.

What I don't understand is why:

• the Notebook has a second environment selector on top of the conda environment selector (to what gain?)
• the Notebook is assumed to work in the same-named environment (which has to be communicated ahead-of-time and breaks using multiple environments for R&D) - who am I to mandate the name of environments on the machines of other folk?
  • If they use their own environment name, modify the Notebook and commit it then I'll get their modified metadata and I'll have to switch it back each time - to what gain?

Does it really make sense to have me activate an environment, open a Notebook (using jupyter in that environment) and then switch to a different environment? Is this a common workflow?

It isn't the biggest hurdle by any means but it seems like an extra step of effort for 0 gain. I'm still left wondering "what am I missing?".

[jankatins]

the Notebook is assumed to work in the same-named environment (which has to be communicated ahead-of-time and breaks using multiple environments for R&D) - who am I to mandate the name of environments on the machines of other folk?

IMO this Assumtion never holds, no matter what you use: In your way, the implicit assumption is that the "default kernel" has all the package which the notebook needs, but that probably also never the case (at least it wouldn't be the case if you would send me a notebook...). And then add different python versions into the mix (e.g. in your case start with py27 environment and then with a py3x env). Either you mandate conventions or you will have these problems.

Does it really make sense to have me activate an environment, open a Notebook (using jupyter in that environment) and then switch to a different environment? Is this a common workflow?

You don't need to activate the notebook env, you just need to call the jupyter script (jupyter.exeon windows) with the full path.

This is my workflow and for that workflow, a kernel manager which adds a kernel for each env is a great help:

• have one env with only the notebook (or use the root env), a env for each project and a project directory which holds the project specific files (=project git repo)
• start a server in project dir with <path to jupyter.exe in notebook env>\jupyter.exe notebook -> notebook server which shows the root dir of the project folder
• create a new notebook by explicitly choosing the project kernel

-> from then on, I just have to just open the notebook and it works

If I would collaborate with someone, I would add a environment.yml into the project folder (=repo) which would hold the name of the env and all packages and then would just ensure that each project member would have an easy command to create the project environment (make environment?). If someone wants to register their environment manually, that's also only one make register_environment away.

Only if someone does not want to use such a env based setup, you are going to need to use the default kernel (e.g. activate the project env, call jupyter-notebook and chose the default kernel ('Python [default]'). Or use a precommit hook / filter to rewrite the kernel name in the ipynb file (I anyway only commit without the numbers and output: http://janschulz.github.io/windows-dev-environment.html -> "proper diffs and commits for notebooks")

It isn't the biggest hurdle by any means but it seems like an extra step of effort for 0 gain

My gain was that I didn't need to manage different kernels manually (the kernelspec install step for each env) and at the same time could use different kernels (=different languages + python versions) in the same project (something you miss when you just have the default kernel available due to just activate .. + jupyter notebook in the project dir).

[maling6154]

@JanSchulz Hey, bro, May I ask you something? I built a separate environment for running tensor flow using python=3.5, and like you said, by using nb_conda_kernels, no matter in which environment to launch jupyter, I will always have different environment kernel. And in every environment, I will have a default kernel, which should be the python or other language I install in my environment, right? So, in my root environment, if I launch jupyter notebook, I will see kernel options, like, default, root, other_environment_kernel. My question is that in my root environment, python is 2.7, but why do I still can type, print(2) in that kernel? This drives me crazy... And another problem is that I have R kernel installed in my root environment, but I can not trigger this kernel at other environment. The whole kernel stuff is driving me to death, I have not even had lunch...Thank you so much if you can help.

[damianavila]

My question is that in my root environment, python is 2.7, but why do I still can type, print(2) in that kernel?

What's the output of sys.executable in that "root" kernel you are mentioning? That will tell you which python are you pointing at...

And another problem is that I have R kernel installed in my root environment, but I can not trigger this kernel at other environment.

You need to have R and the R-kernel installed in that environment so nb_conda_kernels can detect it and create the proper kernelspec...

[damianavila]

@jhprinz can you please open another issue about this? Just to not distort the previous discussion.
In short, you probably need to deal with nb_conda_kernels internals.

[jhprinz]

@damianavila Sure, will just copy this to a new issue. It is kind of a question about the docs though. I just wasn't sure if an issue is the best way to ask for help at all. Would stackoverflow be an alternative?

[damianavila]

@jhprinz I think an issue is good beginning, you can also free to post in stackoverflow but I am not sure if you will find people there to ask this questions.

[damianavila]

Closing this one since the original discussion stopped a long time ago. Please reopen (or open a new issues) if you have further questions.