Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip doesn't work from statically linked executables #6543

Closed
indygreg opened this issue May 26, 2019 · 5 comments

Comments

@indygreg
Copy link

commented May 26, 2019

Environment

  • pip version: 19
  • Python version: 3.7.3 compiled with MUSL
  • OS: Linux

Description

In the bowels of pip, the ctypes module is imported. Importing this module on statically linked executables (such as Python builds compiled with MUSL) fails with OSError because executing ctypes/__init__.py attempts to call dlopen(), which will always fail on binaries that don't support dynamic loading.

This is arguably a bug in CPython's ctypes module, as it could possibly gracefully fail on failure calling dlopen(). But that bug has shipped for years and it is everyone else's responsibility to work around it.

Expected behavior

I think pip should handle failure to import the ctypes module gracefully and not abort.

How to Reproduce

With a statically linked Python executable that doesn't have a .dynamic or other related ELF sections, attempt to run python -m ensurepip install <package>.

The zstd compressed tarball at https://github.com/indygreg/python-build-standalone/releases/download/20190505/cpython-3.7.3-linux64-musl-20190526T0219.tar.zst contains such an executable under python/install/bin/python3.7.

Output

$ python/install/bin/python3.7 -m ensurepip install pyflakes
Traceback (most recent call last):
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/__init__.py", line 40, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/main_parser.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/__init__.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/completion.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/base_command.py", line 20, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/download.py", line 37, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/utils/glibc.py", line 3, in <module>
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ctypes/__init__.py", line 444, in <module>
    pythonapi = PyDLL(None)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported

The problem with CPython specifically:

$ ldd python/install/bin/python3.7
        not a dynamic executable

$ python/install/bin/python3.7
Python 3.7.3 (default, May 26 2019, 04:33:50)
[Clang 7.0.1 (tags/RELEASE_701/final)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gps/src/python-build-standalone.git/build/python/install/lib/python3.7/ctypes/__init__.py", line 444, in <module>
    pythonapi = PyDLL(None)
  File "/home/gps/src/python-build-standalone.git/build/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported
@indygreg

This comment has been minimized.

Copy link
Author

commented May 27, 2019

I filed a CPython issue at https://bugs.python.org/issue37060.

indygreg added a commit to indygreg/pip that referenced this issue May 27, 2019

Handle ImportError and OSError when importing ctypes (pypa#6543)
Non-dynamic executables can raise OSError when importing ctypes
because dlopen(NULL) is called on module import and dlopen()
won't work on non-dynamic executables.

This commit teaches the glibc version sniffing module to
handle a missing or not working ctypes module.

indygreg added a commit to indygreg/pip that referenced this issue May 27, 2019

Handle ImportError and OSError when importing ctypes (pypa#6543)
Non-dynamic executables can raise OSError when importing ctypes
because dlopen(NULL) is called on module import and dlopen()
won't work on non-dynamic executables.

This commit teaches the glibc version sniffing module to
handle a missing or not working ctypes module.
@indygreg

This comment has been minimized.

Copy link
Author

commented May 27, 2019

setuptools is also affected. I figure I'll wait for someone from the PyPA to comment on matters before I file an issue / submit a PR to setuptools.

cpython-linux64-musl> Traceback (most recent call last):
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setup.py", line 11, in <module>
cpython-linux64-musl>     import setuptools
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setuptools/__init__.py", line 20, in <module>
cpython-linux64-musl>     from setuptools.dist import Distribution, Feature
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setuptools/dist.py", line 35, in <module>
cpython-linux64-musl>     from setuptools import windows_support
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setuptools/windows_support.py", line 2, in <module>
cpython-linux64-musl>     import ctypes
cpython-linux64-musl>   File "/build/out/python/install/lib/python3.7/ctypes/__init__.py", line 444, in <module>
cpython-linux64-musl>     pythonapi = PyDLL(None)
cpython-linux64-musl>   File "/build/out/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__
cpython-linux64-musl>     self._handle = _dlopen(self._name, mode)
cpython-linux64-musl> OSError: Dynamic loading not supported
@cjerdonek

This comment has been minimized.

Copy link
Member

commented May 27, 2019

I would go ahead and file an issue on the setuptools tracker as you'll get a different set of readers there.

@pfmoore

This comment has been minimized.

Copy link
Member

commented May 27, 2019

Note - your actual issue here is in pip/_internal/utils/glibc.py and seems relatively shallow - you've posted #6543 which seems to cover it by deciding that if you don't have ctypes, you're not glibc. With that in mind, it's entirely reasonable to ignore the broader question of ctypes on statically linked systems, and just fix that one issue and stop there.

But assuming you do want to treat the broader issue as worth addressing:

I strongly believe that this is a ctypes bug - there's no way that it's reasonable to expect every import of a stdlib module to be checked for errors in every project. And it's not as if ctypes is documented as an optional stdlib module (at least not that I could see). Having said that, I see the reasoning that we have to live with the reality, and therefore we might have to deal with this. But many of our ctypes usages are in vendored libraries (colorama, urllib3) so you'll need to raise issues against those projects as well. And I'd still argue for fixing issues on a case by case basis, prompted by real world problems, rather than just making a blanket rule that we always have to protect ctypes usage (it's not like we make provision for systems where threading isn't available, for example).

I would want to watch the Python bug report closely, though - if the CPython core devs don't accept that it's a bug for "import ctypes" to fail, then at a minimum the CPython docs should make that clear - there's nothing in the docs at the moment to suggest that it's any more acceptable for "import ctypes" to fail than (say) "import subprocess".

indygreg added a commit to indygreg/python-build-standalone that referenced this issue May 27, 2019

linux: install patched versions of setuptools and pip
This is related to https://bugs.python.org/issue37060 and
pypa/pip#6543.

With patched versions of setuptools and pip installed, we can
now `pip install` without issue on non-dynamic Python
executables!
@indygreg

This comment has been minimized.

Copy link
Author

commented May 27, 2019

I agree that this is a ctypes bug. I think it is best for ctypes import to gracefully handle the OSError and continue with import.

Then the question becomes what should consumers of ctypes do - if anything - to work around the import time failure in existing Python releases. I highly doubt many people out there have built a non-dynamic Python binary. I think it is a reasonable stance to tell people they need a minimum version of Python - perhaps 3.7.4 - to run a fully static binary.

But there's still the run-time issue of OSError when calling ctypes.CDLL(None). Presumably pip, setuptools, and the rest of the world will need to be aware that ctypes.CDLL(None) can fail. That's of course a different patch from what I wrote. Perhaps it is best to wait for upstream to weigh in about what ctypes should do before we land any patches.

In any case, I've worked around this issue in python-build-standalone by patching cpython, setuptools, and pip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.