Description
Environment
OS: Ubuntu 16.04 (nvidia/cuda:8.0-cudnn6-devel)
Python version: 3.6.5
Conda version:
conda 4.5.10
Pip version:
pip 18.0
Description
Pip install stopped working during docker build of a complex docker container (based on Kaggle/docker-python) and it cannot be repaired by simple methods such as reinstalling / downgrading with pip itself, with conda or a Miniconda script, unless you change the default installation location (/opt/conda) to an alternative folder when using Minconda script to re-install pip (effectively moving all subsequent libraries installs with pip to a separate location). Note that pip uninstall still works (and can be used to remove pip - apparently not effectively enough).
Expected behavior
pip install should be working even in these conditions, i.e. . pip install scipy should succeed regardless of the complexity of the installation.
How to Reproduce
I managed to make it reproducible by providing a functional but only partially completed docker container produced by docker build just before the error occured (I verified that it is reproducible on two different servers).
The bug can be reproduced by executing pip install in bash in this archived docker image that captures this bug (no login requried):
docker pull mirekphd/tst-ml-gpu-py36-cuda8-failing:001766c55ca8
docker run -it --rm --name test -u 0 mirekphd/tst-ml-gpu-py36-cuda8-failing:001766c55ca8 bash
pip install scipy
Possible causes
The last completed operation during build and hence the possible culprit was Jupyter Notebook (or Notebook Extensions) installation with conda (the only installer that makes Jupyter Notebook Extensions work in Jupyter Notebook - pip installation made earlier did not suffice). When I moved Jupyter (Notebook, Lab and Notebook Extensions) installation to the end of the build process (i.e. after all other libraries), the build completed and all libraries can be loaded (from a single standard location in /opt/conda). However, Jupyter section was once again the last one to succeed with pip install, and pip still gets broken, raising the same error (which can be even considered a feature not a bug as it helps maintain library / ML models reproducibility:). I also notice that the same error is raised by pip show and pipdeptree (e.g. !pipdeptree --reverse --packages xgboost) but not by pip freeze.
An intial workaround
A complex but universal workaround I found was to change the pip installation location into an alternative folder when using Miniconda script to re-install pip.
Dockerfile excerpt that repairs this issue and allows the build to succeed:
# ARG MINICONDA_VER=4.5.4 # (same as defined above)
# ARG PIP_VER=18.0 # (same as defined above)
# define alternative pip location
ENV PIP_RESCUE_FOLDER=/tmp/opt/conda
RUN mkdir -p ${PIP_RESCUE_FOLDER} && cd ${PIP_RESCUE_FOLDER} && \
# download Miniconda installation script including pip binary
wget https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VER}-Linux-x86_64.sh && \
# note we install pip (and entire Miniconda) in the secondary location by specifying the "prefix folder",
# using parameter -p (hence PREFIX=/tmp/opt/conda and not /opt/conda as it is usually done)
/bin/bash Miniconda3-${MINICONDA_VER}-Linux-x86_64.sh -f -b -p ${PIP_RESCUE_FOLDER} && \
# /tmp/opt/conda/bin/pip --version
# pip 10.0.1 from /tmp/opt/conda/lib/python3.6/site-packages/pip (python 3.6)
# update secondary pip to the desired version
/tmp/opt/conda/bin/pip install --upgrade pip==${PIP_VER} && \
# overwrite pip in primary location with the one just installed into secondary one
cp /tmp/opt/conda/bin/pip /opt/conda/bin && \
cp -R /tmp/opt/conda/lib/python3.6/site-packages/pip /opt/conda/lib/python3.6/site-packages
# to make the libraries installed by secondary pip (into the pip rescue folder) work in python
# we need to add this secondary bin folder to PATH
ENV PATH=$PIP_RESCUE_FOLDER/bin:$PATH
# to verify it works:
# $python
# >>> import openshift as oc
# >>> oc
# <module 'openshift' from '/tmp/opt/conda/lib/python3.6/site-packages/openshift/__init__.py'>
Output
The expected error message is:
root@2dba6ccb6b94:/# pip install scipy --no-cache-dir
Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (1.1.0)
Requirement already satisfied: numpy>=1.8.2 in /opt/conda/lib/python3.6/site-packages (from scipy) (1.15.2)
Exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2869, in _dep_map
return self.__dep_map
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2663, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 93, in __init__
req = REQUIREMENT.parseString(requirement_string)
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1632, in parseString
raise exc
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1622, in parseString
loc, tokens = self._parse( instring, 0 )
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1379, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3395, in parseImpl
loc, exprtokens = e._parse( instring, loc, doActions )
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 1383, in _parseNoCache
loc,tokens = self.parseImpl( instring, preloc, doActions )
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pyparsing.py", line 3183, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 33), (line:1, col:34)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2949, in __init__
super(Requirement, self).__init__(requirement_string)
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/packaging/requirements.py", line 97, in __init__
requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'; extra '"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 141, in main
status = self.run(options, args)
File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 330, in run
self._warn_about_conflicts(to_install)
File "/opt/conda/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 456, in _warn_about_conflicts
package_set, _dep_info = check_install_conflicts(to_install)
File "/opt/conda/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 98, in check_install_conflicts
package_set = create_package_set_from_installed()
File "/opt/conda/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 41, in create_package_set_from_installed
package_set[name] = PackageDetails(dist.version, dist.requires())
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2607, in requires
dm = self._dep_map
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2871, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2881, in _compute_dependencies
reqs.extend(parse_requirements(req))
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2942, in parse_requirements
yield Requirement(line)
File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2951, in __init__
raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'; extra '"
root@2dba6ccb6b94:/#