_pip install_ broken beyond normal repair by Jupyter Notebook Extensions (AttributeError: _DistInfoDistribution__dep_map)

[mirekphd]

Environment

OS: Ubuntu 16.04 (nvidia/cuda:8.0-cudnn6-devel)

Python version: 3.6.5

Conda version:
conda 4.5.10

Pip version:
pip 18.0

Description
Pip install stopped working during docker build of a complex docker container (based on Kaggle/docker-python) and it cannot be repaired by simple methods such as reinstalling / downgrading with pip itself, with conda or a Miniconda script, unless you change the default installation location (/opt/conda) to an alternative folder when using Minconda script to re-install pip (effectively moving all subsequent libraries installs with pip to a separate location). Note that pip uninstall still works (and can be used to remove pip - apparently not effectively enough).

Expected behavior
pip install should be working even in these conditions, i.e. . pip install scipy should succeed regardless of the complexity of the installation.

How to Reproduce
I managed to make it reproducible by providing a functional but only partially completed docker container produced by docker build just before the error occured (I verified that it is reproducible on two different servers).

The bug can be reproduced by executing pip install in bash in this archived docker image that captures this bug (no login requried):

docker pull mirekphd/tst-ml-gpu-py36-cuda8-failing:001766c55ca8
docker run -it --rm --name test -u 0 mirekphd/tst-ml-gpu-py36-cuda8-failing:001766c55ca8 bash
pip install scipy

Possible causes
The last completed operation during build and hence the possible culprit was Jupyter Notebook (or Notebook Extensions) installation with conda (the only installer that makes Jupyter Notebook Extensions work in Jupyter Notebook - pip installation made earlier did not suffice). When I moved Jupyter (Notebook, Lab and Notebook Extensions) installation to the end of the build process (i.e. after all other libraries), the build completed and all libraries can be loaded (from a single standard location in /opt/conda). However, Jupyter section was once again the last one to succeed with pip install, and pip still gets broken, raising the same error (which can be even considered a feature not a bug as it helps maintain library / ML models reproducibility:). I also notice that the same error is raised by pip show and pipdeptree (e.g. !pipdeptree --reverse --packages xgboost) but not by pip freeze.

An intial workaround
A complex but universal workaround I found was to change the pip installation location into an alternative folder when using Miniconda script to re-install pip.

Dockerfile excerpt that repairs this issue and allows the build to succeed:

# ARG MINICONDA_VER=4.5.4 # (same as defined above)
# ARG PIP_VER=18.0 # (same as defined above)
# define alternative pip location
ENV PIP_RESCUE_FOLDER=/tmp/opt/conda

RUN mkdir -p ${PIP_RESCUE_FOLDER} && cd ${PIP_RESCUE_FOLDER} && \
	# download Miniconda installation script including pip binary
	wget https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VER}-Linux-x86_64.sh && \
	# note we install pip (and entire Miniconda) in the secondary location by specifying the "prefix folder",
	# using parameter -p (hence PREFIX=/tmp/opt/conda and not /opt/conda as it is usually done)
	/bin/bash Miniconda3-${MINICONDA_VER}-Linux-x86_64.sh -f -b -p ${PIP_RESCUE_FOLDER} && \
	# /tmp/opt/conda/bin/pip --version
	# pip 10.0.1 from /tmp/opt/conda/lib/python3.6/site-packages/pip (python 3.6)
	# update secondary pip to the desired version
	/tmp/opt/conda/bin/pip install --upgrade pip==${PIP_VER} && \
	# overwrite pip in primary location with the one just installed into secondary one
	cp /tmp/opt/conda/bin/pip /opt/conda/bin && \
	cp -R /tmp/opt/conda/lib/python3.6/site-packages/pip /opt/conda/lib/python3.6/site-packages

# to make the libraries installed by secondary pip (into the pip rescue folder) work in python
# we need to add this secondary bin folder to PATH
ENV PATH=$PIP_RESCUE_FOLDER/bin:$PATH
# to verify it works:
# $python
# >>> import openshift as oc
# >>> oc
# <module 'openshift' from '/tmp/opt/conda/lib/python3.6/site-packages/openshift/__init__.py'>

Output

The expected error message is:

root@2dba6ccb6b94:/# pip install scipy --no-cache-dir
Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (1.1.0)
Requirement already satisfied: numpy>=1.8.2 in /opt/conda/lib/python3.6/site-packages (from scipy) (1.15.2)
Exception:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2869, in _dep_map
    return self.__dep_map
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2663, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1632, in parseString
    raise exc
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1622, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3395, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1383, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3183, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 33), (line:1, col:34)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2949, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 97, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'; extra '"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 141, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 330, in run
    self._warn_about_conflicts(to_install)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 456, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 98, in check_install_conflicts
    package_set = create_package_set_from_installed()
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 41, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2607, in requires
    dm = self._dep_map
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2871, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2881, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2942, in parse_requirements
    yield Requirement(line)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2951, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'; extra '"
root@2dba6ccb6b94:/# 

[mirekphd]

The culprit that broke pip install was not Jupyter Notebook itself (already installed successfully using pip earlier in the build sequence), but its Extensions / Configurator installed with conda (!) in the versions specified below:

# Jupyter notebook extensions
# caution: Jupyter notebook has to be pinned, because Jupyter notebook extensions 
# not always work with latest versions of notebook
# pin Jupyter Notebook
ARG NB_VER=5.6.0                    
# pin Jupyter notebook extensions
ARG NB_EXT_VER=0.5.0
# pin Jupyter notebook extensions configurator
ARG NB_EXT_CFG_VER=0.4.0

# caution: jupyter notebook extensions do not work unless installed with conda (pip does not suffice)
RUN conda install -c conda-forge notebook=${NB_VER} \ 
jupyter_contrib_nbextensions=${NB_EXT_VER} \ 
jupyter_nbextensions_configurator=${NB_EXT_CFG_VER} 

Note that not installing these extensions (which prevents pip from breaking down) is not the same as simply uninstalling them (which does not work - if these extensions are installed even once, pip gets broken beyond repair). If that breakage happened outside boundaries of an isolated container environment, it would be pretty serious.

[takluyver]

It looks like #3704 fixed a similar issue for pip freeze, but this error is coming from the new functionality that checks all installed packages on every pip install. I think a similar try/except is needed around this line:

pip/src/pip/_internal/operations/check.py

Line 41 in b6bbabe

package_set[name] = PackageDetails(dist.version, dist.requires())

I'd also say that the whole check operation should be try/excepted, because bugs in that shouldn't cause pip install to blow up. I think this would be the relevant line:

pip/src/pip/_internal/commands/install.py

Line 475 in b6bbabe

package_set, _dep_info = check_install_conflicts(to_install)

I think it's a bug in Flit causing at least some of these problems - sorry about that, and hopefully I've got it fixed now. But clearly an installed package with broken metadata shouldn't break pip like this.

[takluyver]

I've had a go at fixing this in #5842 .

[astrojuanlu]

I've found this particular issue in older versions of pip as well, including 10 and 9.