Skip to content

_pip install_ broken beyond normal repair by Jupyter Notebook Extensions (AttributeError: _DistInfoDistribution__dep_map) #5839

Closed
@mirekphd

Description

@mirekphd

Environment

OS: Ubuntu 16.04 (nvidia/cuda:8.0-cudnn6-devel)

Python version: 3.6.5

Conda version:
conda 4.5.10

Pip version:
pip 18.0

Description
Pip install stopped working during docker build of a complex docker container (based on Kaggle/docker-python) and it cannot be repaired by simple methods such as reinstalling / downgrading with pip itself, with conda or a Miniconda script, unless you change the default installation location (/opt/conda) to an alternative folder when using Minconda script to re-install pip (effectively moving all subsequent libraries installs with pip to a separate location). Note that pip uninstall still works (and can be used to remove pip - apparently not effectively enough).

Expected behavior
pip install should be working even in these conditions, i.e. . pip install scipy should succeed regardless of the complexity of the installation.

How to Reproduce
I managed to make it reproducible by providing a functional but only partially completed docker container produced by docker build just before the error occured (I verified that it is reproducible on two different servers).

The bug can be reproduced by executing pip install in bash in this archived docker image that captures this bug (no login requried):

docker pull mirekphd/tst-ml-gpu-py36-cuda8-failing:001766c55ca8
docker run -it --rm --name test -u 0 mirekphd/tst-ml-gpu-py36-cuda8-failing:001766c55ca8 bash
pip install scipy

Possible causes
The last completed operation during build and hence the possible culprit was Jupyter Notebook (or Notebook Extensions) installation with conda (the only installer that makes Jupyter Notebook Extensions work in Jupyter Notebook - pip installation made earlier did not suffice). When I moved Jupyter (Notebook, Lab and Notebook Extensions) installation to the end of the build process (i.e. after all other libraries), the build completed and all libraries can be loaded (from a single standard location in /opt/conda). However, Jupyter section was once again the last one to succeed with pip install, and pip still gets broken, raising the same error (which can be even considered a feature not a bug as it helps maintain library / ML models reproducibility:). I also notice that the same error is raised by pip show and pipdeptree (e.g. !pipdeptree --reverse --packages xgboost) but not by pip freeze.

An intial workaround
A complex but universal workaround I found was to change the pip installation location into an alternative folder when using Miniconda script to re-install pip.

Dockerfile excerpt that repairs this issue and allows the build to succeed:

# ARG MINICONDA_VER=4.5.4 # (same as defined above)
# ARG PIP_VER=18.0 # (same as defined above)
# define alternative pip location
ENV PIP_RESCUE_FOLDER=/tmp/opt/conda

RUN mkdir -p ${PIP_RESCUE_FOLDER} && cd ${PIP_RESCUE_FOLDER} && \
	# download Miniconda installation script including pip binary
	wget https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VER}-Linux-x86_64.sh && \
	# note we install pip (and entire Miniconda) in the secondary location by specifying the "prefix folder",
	# using parameter -p (hence PREFIX=/tmp/opt/conda and not /opt/conda as it is usually done)
	/bin/bash Miniconda3-${MINICONDA_VER}-Linux-x86_64.sh -f -b -p ${PIP_RESCUE_FOLDER} && \
	# /tmp/opt/conda/bin/pip --version
	# pip 10.0.1 from /tmp/opt/conda/lib/python3.6/site-packages/pip (python 3.6)
	# update secondary pip to the desired version
	/tmp/opt/conda/bin/pip install --upgrade pip==${PIP_VER} && \
	# overwrite pip in primary location with the one just installed into secondary one
	cp /tmp/opt/conda/bin/pip /opt/conda/bin && \
	cp -R /tmp/opt/conda/lib/python3.6/site-packages/pip /opt/conda/lib/python3.6/site-packages

# to make the libraries installed by secondary pip (into the pip rescue folder) work in python
# we need to add this secondary bin folder to PATH
ENV PATH=$PIP_RESCUE_FOLDER/bin:$PATH
# to verify it works:
# $python
# >>> import openshift as oc
# >>> oc
# <module 'openshift' from '/tmp/opt/conda/lib/python3.6/site-packages/openshift/__init__.py'>

Output

The expected error message is:

root@2dba6ccb6b94:/# pip install scipy --no-cache-dir
Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (1.1.0)
Requirement already satisfied: numpy>=1.8.2 in /opt/conda/lib/python3.6/site-packages (from scipy) (1.15.2)
Exception:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2869, in _dep_map
    return self.__dep_map
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2663, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1632, in parseString
    raise exc
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1622, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1379, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3395, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1383, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3183, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 33), (line:1, col:34)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2949, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 97, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'; extra '"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 141, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 330, in run
    self._warn_about_conflicts(to_install)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 456, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 98, in check_install_conflicts
    package_set = create_package_set_from_installed()
  File "/opt/conda/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 41, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2607, in requires
    dm = self._dep_map
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2871, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2881, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2942, in parse_requirements
    yield Requirement(line)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2951, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'; extra '"
root@2dba6ccb6b94:/# 

Metadata

Metadata

Assignees

No one assigned

    Labels

    C: checkChecking dependency graph for consistencyauto-lockedOutdated issues that have been locked by automationtype: bugA confirmed bug or unintended behavior

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions