pip install ipykernel<6 gives incomplete installation, since Traitlets 5.1 release

[mdickinson]

With the release of Traitlets 5.1, it appears that an ipykernel installed with pip install 'ipykernel < 6' isn't immediately usable: ipython_genutils is not installed into the environment, and an import ipykernel gives a ModuleNotFoundError: No module named 'ipython_genutils'. It looks as though ipykernel depends on ipython_genutils and is currently relying on the traitlets dependencies to ensure that ipython_genutils is installed, but since Traitlets 5.1, ipython_genutils is no longer a dependency of traitlets. Moreover, explicitly installing ipython_genutils seems like the wrong solution, since the PyPI page seems to instruct us that we shouldn't need to know about ipython_genutils. :-)

It's not clear to me whether this should be classified as a traitlets issue, an ipykernel issue, or a non-issue; it was the Traitlets release that triggered the issue, but presumably if it's deemed worth fixing, then fixing in the 5.x maintenance branch here (e.g., by adding ipython_genutils as an explicit dependency) makes most sense. Apologies if I've opened this in the wrong place.

Steps to reproduce in a clean Python 3.9 venv:

• Create a fresh Python 3.9 venv, and activate that environment
• Do a pip install 'ipykernel < 6'
• Start Python and try an import ipykernel. The expectation is that this would succeed, but it fails with a ModuleNotFoundError.

Here are the results on my (macOS 10.15.7) machine.

mdickinson@mirzakhani Desktop % python -m venv --clear testing && source testing/bin/activate
(testing) mdickinson@mirzakhani Desktop % pip install 'ipykernel<6'
Collecting ipykernel<6
  Using cached ipykernel-5.5.5-py3-none-any.whl (120 kB)
Collecting appnope
  Using cached appnope-0.1.2-py2.py3-none-any.whl (4.3 kB)
Collecting jupyter-client
  Using cached jupyter_client-7.0.2-py3-none-any.whl (122 kB)
Collecting ipython>=5.0.0
  Using cached ipython-7.27.0-py3-none-any.whl (787 kB)
Collecting tornado>=4.2
  Using cached tornado-6.1-cp39-cp39-macosx_10_9_x86_64.whl (416 kB)
Collecting traitlets>=4.1.0
  Using cached traitlets-5.1.0-py3-none-any.whl (101 kB)
Collecting backcall
  Using cached backcall-0.2.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools>=18.5 in ./testing/lib/python3.9/site-packages (from ipython>=5.0.0->ipykernel<6) (57.4.0)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Using cached prompt_toolkit-3.0.20-py3-none-any.whl (370 kB)
Collecting pygments
  Using cached Pygments-2.10.0-py3-none-any.whl (1.0 MB)
Collecting matplotlib-inline
  Using cached matplotlib_inline-0.1.2-py3-none-any.whl (8.2 kB)
Collecting pexpect>4.3
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting jedi>=0.16
  Using cached jedi-0.18.0-py2.py3-none-any.whl (1.4 MB)
Collecting decorator
  Using cached decorator-5.0.9-py3-none-any.whl (8.9 kB)
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting parso<0.9.0,>=0.8.0
  Using cached parso-0.8.2-py2.py3-none-any.whl (94 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting python-dateutil>=2.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting entrypoints
  Using cached entrypoints-0.3-py2.py3-none-any.whl (11 kB)
Collecting jupyter-core>=4.6.0
  Using cached jupyter_core-4.7.1-py3-none-any.whl (82 kB)
Collecting pyzmq>=13
  Using cached pyzmq-22.2.1-cp39-cp39-macosx_10_15_universal2.whl (2.0 MB)
Collecting nest-asyncio>=1.5
  Using cached nest_asyncio-1.5.1-py3-none-any.whl (5.0 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: wcwidth, traitlets, six, ptyprocess, parso, tornado, pyzmq, python-dateutil, pygments, prompt-toolkit, pickleshare, pexpect, nest-asyncio, matplotlib-inline, jupyter-core, jedi, entrypoints, decorator, backcall, appnope, jupyter-client, ipython, ipykernel
Successfully installed appnope-0.1.2 backcall-0.2.0 decorator-5.0.9 entrypoints-0.3 ipykernel-5.5.5 ipython-7.27.0 jedi-0.18.0 jupyter-client-7.0.2 jupyter-core-4.7.1 matplotlib-inline-0.1.2 nest-asyncio-1.5.1 parso-0.8.2 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-3.0.20 ptyprocess-0.7.0 pygments-2.10.0 python-dateutil-2.8.2 pyzmq-22.2.1 six-1.16.0 tornado-6.1 traitlets-5.1.0 wcwidth-0.2.5
WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/mdickinson/Desktop/testing/bin/python -m pip install --upgrade pip' command.
(testing) mdickinson@mirzakhani Desktop % python
Python 3.9.7 (default, Sep  1 2021, 04:00:27) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipykernel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mdickinson/Desktop/testing/lib/python3.9/site-packages/ipykernel/__init__.py", line 2, in <module>
    from .connect import *
  File "/Users/mdickinson/Desktop/testing/lib/python3.9/site-packages/ipykernel/connect.py", line 13, in <module>
    from ipython_genutils.path import filefind
ModuleNotFoundError: No module named 'ipython_genutils'
>>> 

For context: we have some code that works closely with ipykernel. That code currently works with ipykernel 5, but it will take some effort to make it compatible with ipykernel 6. In the meantime, we put an ipykernel < 6 version restriction in place, and then ran into this issue. This isn't an issue for us: we already have a workaround in place (adding traitlets < 5.1 to our dependencies), but it seemed worth reporting in case it affects others.

[Carreau]

ipykernel 6.3.1 was released yesterday I believe that list ipython_genutils as a dependency. it is identical to 6.3.0 otherwise.
Sorry for the missing dependency. In the meantime if you can't be on 6.3.1 I think that manually adding ipython_genutils is a good workaround.

We'll finish removing ipython_genutils from ipykernel in the future.

[mdickinson]

Thanks, @Carreau. If you're planning to make a 5.x bugfix release at any point then this seems like something that could go into that bugfix release (and I can make a PR if that helps), but I'm assuming that all the effort is focused on the 6.x development at this point. In which case this should probably be closed.

[Carreau]

Unless there is a strong reason for people to keep 5.x around I doubt we'll make another release, but it's still possible.

Are you sill using 5.x because you are on an older Python ?

[frenzymadness]

The reason to install ipykernel < 6 is that envisage project is not yet compatible with ipykernel 6. For more info: enthought/envisage#448

[carlosperate]

The Mu Editor uses ipykernel < 6 to maintain compatibility with macOS 10.13 High Sierra (and older versions), as v6 adds the debugpy dependency, which only has wheels only for 10.14+ in its earliest release, and it's even moved to 10.15+ in the latest releases.

And we also support older Python versions than 3.7. Even if are soon to drop support for 3.5, 3.6 is still the default version in Ubuntu 18.04.

So a 5.x update would be quite welcomed.

[DonJayamanne]

I'm running into a related issue.
When installing ipykernel on Mac m1 using conda conda install ipykernel, the latest version installed is 5.3.4, hence even with the fix to include ipython_genutils, we still end up with ipykernel==5.3.4 which doesn't fix it.
Shouldn't we remove the 5.3.4 version from arm64 channel,
I.e. in the main default (main) channel the only version of ipykernle is 5.3.4 for Mac arm64 (Mac m1).
https://repo.anaconda.com/pkgs/main/osx-arm64/ipykernel-5.3.4-py39h33ce5c2_0.tar.bz2

Happy to file a separate bug if this is unrelated.

[blink1073]

I can cut a release of 5.x if someone is able to make a PR with the added ipython_genutils dependency.

@DonJayamanne, I think that is a bug to report to Anaconda, to ask them to publish the forthcoming version of ipykernel 5.x

[blink1073]

Fixed in #778, thanks again @DonJayamanne!

https://github.com/ipython/ipykernel/releases/tag/5.5.6