KeyError: '.libs' when installing from repaired wheel

[vsalvino]

Environment

• pip version: 19.2.3
• Python version: 3.7.4
• OS: Docker image: python:3.7.4-slim-stretch

Description
Not sure if this is a problem with pip, auditwheel, or the python package. But here goes:

When trying to install a wheel that was build from source, and then patched with auditwheel repair, pip install fails with the following error:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 407, in run
    use_user_site=options.use_user_site,
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 928, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 461, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/wheel.py", line 463, in move_wheel_files
    dest = scheme[subdir]
KeyError: '.libs'

Expected behavior
The wheel should install as expected.

How to Reproduce
The premise is to build a wheel on a specific version of linux/python, patch the wheel with necessary system libs, and then try to install patched wheel on an identical system (without the build dependencies). This process works fine on package mysqlclient, however the same process produces the error above for package uwsgi. I'm not sure if this is a problem with pip, auditwheel, or uwsgi. However un-repaired wheels install correctly (but don't work obviously due to missing libs).

Dockerfile to reproduce:

FROM python:3.7.4-slim-stretch as wheelmaker

RUN set -eux; \
    # Install system depencendies
    apt-get update && apt-get install -y --no-install-recommends debhelper libmariadbclient-dev gcc g++ unzip wget; \
    # Compile and install latest version of patchelf for auditwheel
    wget https://github.com/NixOS/patchelf/archive/0.10.zip; \
    unzip 0.10.zip; \
    cd patchelf-0.10; ./bootstrap.sh; ./configure; make; make install; \
    # Build wheels from source
    mkdir /tmp/wheels/; \
    pip install auditwheel; \
    pip wheel --wheel-dir /tmp/wheels/ --no-binary all mysqlclient uwsgi; \
    # Patch wheels with necessary linked binaries
    mkdir /tmp/wheelhouse/; \
    for whl in $(find /tmp/wheels/ -name "*.whl"); do \
        auditwheel repair --wheel-dir /tmp/wheelhouse/ --plat linux_x86_64 $whl; \
    done;


# Clean python image to install our wheels without all the build cruft.
FROM python:3.7.4-slim-stretch

# Copy wheels from builder
COPY --from=wheelmaker /tmp/wheelhouse/ /tmp/wheelhouse/

RUN set -eux; \
    # Install wheels
    pip install $(find /tmp/wheelhouse/ -name "*.whl"); \
    # Installs mysqlclient correctly, but throws KeyError when trying to copy libs from uwsgi
    # Same error applies when installing on same build machine, different machine, etc.

I have inspected the wheel files manually by unzipping them; everything looks correct, the .lib directory is in place as expected contents appear to match entries in RECORDS; etc. This all leads me to believe the problem is in pip and not in auditwheel or the python package itself. I have also tried manually running the steps above on different systems, with the same result.

I would be happy to troubleshoot further if someone with more knowledge of pip could provide some ideas.

[chrahunt]

Hello! Thanks for the easy reproducer. I built and extracted your wheel and see:

$ find .
.
./uWSGI-2.0.18.data
./uWSGI-2.0.18.data/.libs
./uWSGI-2.0.18.data/.libs/libutil-2-d463186f.24.so
./uWSGI-2.0.18.data/.libs/libm-2-bfba5a97.24.so
./uWSGI-2.0.18.data/.libs/libdl-2-492aff8a.24.so
./uWSGI-2.0.18.data/.libs/libc-2-d37be6b0.24.so
./uWSGI-2.0.18.data/.libs/libpthread-2-55e67a47.24.so
./uWSGI-2.0.18.data/.libs/libcrypt-2-91e9f20b.24.so
./uWSGI-2.0.18.data/.libs/libz-7fd423a0.so.1.2.8
./uWSGI-2.0.18.data/purelib
./uWSGI-2.0.18.data/purelib/uwsgidecorators.py
./uWSGI-2.0.18.data/scripts
./uWSGI-2.0.18.data/scripts/uwsgi
./uWSGI-2.0.18.dist-info
./uWSGI-2.0.18.dist-info/RECORD
./uWSGI-2.0.18.dist-info/top_level.txt
./uWSGI-2.0.18.dist-info/METADATA
./uWSGI-2.0.18.dist-info/WHEEL

Per PEP 427:

... Each subdirectory of distribution-1.0.data/ is a key into a dict of destination directories, such as distribution-1.0.data/(purelib|platlib|headers|scripts|data). The initially supported paths are taken from distutils.command.install.

If you do python -c 'from distutils.command.install import INSTALL_SCHEMES; print(INSTALL_SCHEMES)' then it shows the various directories available. In this case you'll want the data subdirectory, so in the wheel it should be e.g. uWSGI-2.0.18.data/data/.libs/libutil-2-d463186f.24.so instead of uWSGI-2.0.18.data/.libs/libutil-2-d463186f.24.so.

I manually made the modification to the wheel and was able to install it successfully.

[vsalvino]

Thanks for the quick response. I see exactly what you mean; had no idea about the different folder structures within wheels.

Interestingly enough, after fixing the uWSGI wheel as you described, all the files do install correctly, but the program can't seem to find the libs when trying to run uwsgi. That would probably be an issue for auditwheel though, unless you have any additional ideas related to the wheel package itself.

[chrahunt]

Maybe better as a general question for the packaging category on discuss.python.org. You can also find some discussion in this thread.

I'll close this since there isn't any pip action, but please feel free to drop a link here to any further discussions!

[vsalvino]

Thanks for the links, those are very relevant discussions. I'm still digging into why uwsgi is not picking up the libs bundled with auditwheel. Will open a new issue in the relevant place if any more info comes to light.