Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend wheel ABI support for pypy. #222

Closed
guoshimin opened this Issue Mar 16, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@guoshimin
Copy link

guoshimin commented Mar 16, 2016

pypy version:

$ pypy
Python 2.7.10 (246c9cf22037b11dc0e8c29ce3f291d3b8c5935a, Mar 10 2016, 18:28:19)
[PyPy 5.0.0 with GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>>

The following command fails:

$ pex --python=pypy psutil
**** Failed to install psutil-4.1.0. stdout:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.10-x86_64-2.7
creating build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/__init__.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_common.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_compat.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_psbsd.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_pslinux.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_psosx.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_psposix.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_pssunos.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
copying psutil/_pswindows.py -> build/lib.macosx-10.10-x86_64-2.7/psutil
creating build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/__init__.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/runner.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_bsd.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_linux.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_memory_leaks.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_misc.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_osx.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_posix.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_process.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_sunos.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_system.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
copying psutil/tests/test_windows.py -> build/lib.macosx-10.10-x86_64-2.7/psutil/tests
running build_ext
building 'psutil._psutil_osx' extension
creating build/temp.macosx-10.10-x86_64-2.7
creating build/temp.macosx-10.10-x86_64-2.7/psutil
creating build/temp.macosx-10.10-x86_64-2.7/psutil/arch
creating build/temp.macosx-10.10-x86_64-2.7/psutil/arch/osx
cc -arch x86_64 -O2 -fPIC -Wimplicit -DPSUTIL_VERSION=410 -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/_psutil_osx.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.o
cc -arch x86_64 -O2 -fPIC -Wimplicit -DPSUTIL_VERSION=410 -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/_psutil_common.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_common.o
cc -arch x86_64 -O2 -fPIC -Wimplicit -DPSUTIL_VERSION=410 -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/arch/osx/process_info.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/arch/osx/process_info.o
cc -shared -undefined dynamic_lookup -arch x86_64 build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_common.o build/temp.macosx-10.10-x86_64-2.7/psutil/arch/osx/process_info.o -o build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.pypy-41.so -framework CoreFoundation -framework IOKit
building 'psutil._psutil_posix' extension
cc -arch x86_64 -O2 -fPIC -Wimplicit -I/usr/local/Cellar/pypy/5.0.0/libexec/include -c psutil/_psutil_posix.c -o build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.o
cc -shared -undefined dynamic_lookup -arch x86_64 build/temp.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.pypy-41.so
installing to build/bdist.macosx-10.10-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.10-x86_64
creating build/bdist.macosx-10.10-x86_64/wheel
creating build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/__init__.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_common.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_compat.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psbsd.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_pslinux.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psosx.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psposix.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_pssunos.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_osx.pypy-41.so -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_psutil_posix.pypy-41.so -> build/bdist.macosx-10.10-x86_64/wheel/psutil
copying build/lib.macosx-10.10-x86_64-2.7/psutil/_pswindows.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil
creating build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/__init__.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/runner.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_bsd.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_linux.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_memory_leaks.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_misc.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_osx.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_posix.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_process.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_sunos.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_system.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
copying build/lib.macosx-10.10-x86_64-2.7/psutil/tests/test_windows.py -> build/bdist.macosx-10.10-x86_64/wheel/psutil/tests
running install_egg_info
running egg_info
writing psutil.egg-info/PKG-INFO
writing dependency_links to psutil.egg-info/dependency_links.txt
writing top-level names to psutil.egg-info/top_level.txt
reading manifest file 'psutil.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'psutil.egg-info/SOURCES.txt'
Copying psutil.egg-info to build/bdist.macosx-10.10-x86_64/wheel/psutil-4.1.0-py2.7.egg-info
running install_scripts

**** Failed to install psutil-4.1.0. stderr:
warning: no previously-included files matching '*' found under directory 'docs/_build'
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "setup.py", line 284, in <module>
    main()
  File "setup.py", line 281, in main
    setup(**setup_args)
  File "/usr/local/Cellar/pypy/5.0.0/libexec/lib-python/2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/local/Cellar/pypy/5.0.0/libexec/lib-python/2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/pypy/5.0.0/libexec/lib-python/2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "build/bdist.macosx-10.10-x86_64/egg/wheel/bdist_wheel.py", line 213, in run
    archive_basename = self.get_archive_basename()
  File "build/bdist.macosx-10.10-x86_64/egg/wheel/bdist_wheel.py", line 161, in get_archive_basename
    impl_tag, abi_tag, plat_tag = self.get_tag()
  File "build/bdist.macosx-10.10-x86_64/egg/wheel/bdist_wheel.py", line 155, in get_tag
    assert tag == supported_tags[0]
AssertionError

Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 315, in execute
  File ".bootstrap/_pex/pex.py", line 250, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 282, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 358, in _execute
  File ".bootstrap/_pex/pex.py", line 416, in execute_entry
  File ".bootstrap/_pex/pex.py", line 434, in execute_pkg_resources
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/bin/pex.py", line 509, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/bin/pex.py", line 471, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/resolver.py", line 191, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/resolver.py", line 248, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/shimin/.pex/install/pex-1.0.3-py2.py3-none-any.whl.9b479c2967fcb9866cb8b019740e265c9bc00da2/pex-1.0.3-py2.py3-none-any.whl/pex/resolver.py", line 160, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage(u'file:///Users/shimin/.pex/build/psutil-4.1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
@kwlzn

This comment has been minimized.

Copy link
Member

kwlzn commented Mar 16, 2016

pex + pypy + native deps has never been proven to be super reliable, but it seems to me that the error message you've pasted is actually coming from the translation phase where pex is mostly removed from the picture - e.g. the equivalent of:

wget $PSUTIL_URL
tar zxvf $PSUTIL_TARBALL
cd $PSUTIL_DIR
pypy setup.py bdist_wheel

if you manually run the setup.py in this fashion, do you hit the same AssertionError?

does it change if opt for eggs only (--no-wheel)?

for pex + pypy users, we also suggest using --not-zip-safe and --always-write-cache due to https://bitbucket.org/pypy/pypy/issues/1686/issue-with-our-zipimporter-module.

@guoshimin

This comment has been minimized.

Copy link
Author

guoshimin commented Mar 17, 2016

I will try these commands when I get in front of a computer, but just fyi,
pip_pypy wheel psutil works.
On Mar 16, 2016 3:45 PM, "Kris Wilson" notifications@github.com wrote:

pex + pypy + native deps has never been proven to be super reliable, but
it seems to me that the error message you've pasted is actually coming from
the translation phase where pex is mostly removed from the picture - e.g.
the equivalent of:

wget $PSUTIL_URL
tar zxvf $PSUTIL_TARBALL
cd $PSUTIL_DIR
pypy setup.py bdist_wheel

if you manually run the setup.py in this fashion, do you hit the same
AssertionError?

does it change if opt for eggs only (--no-wheel)?

for pex + pypy users, we also suggest using --not-zip-safe and
--always-write-cache due to
https://bitbucket.org/pypy/pypy/issues/1686/issue-with-our-zipimporter-module
.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#222 (comment)

@guoshimin

This comment has been minimized.

Copy link
Author

guoshimin commented Mar 17, 2016

Running setup.py manually works.

pex --python=pypy --no-wheel psutil also works

It also doesn't work if I build the wheel first and then tell pex to use the local wheel.

shimin /tmp$ pip_pypy wheel -w deps psutil
Collecting psutil
  Using cached psutil-4.1.0.tar.gz
Building wheels for collected packages: psutil
  Running setup.py bdist_wheel for psutil ... done
  Stored in directory: /private/tmp/deps
Successfully built psutil
shimin /tmp$ ls deps
psutil-4.1.0-pp250-pypy_41-macosx_10_10_x86_64.whl
shimin /tmp$ pex --python pypy --disable-cache --no-pypi -f ./deps psutil -v -v -v
pex: crawling link i=0 link=Link('file:///private/tmp/deps') follow_links=False
Could not satisfy all requirements for psutil:
    psutil

It seems it's related to #213. Looking at this code:

for p in platforms + ['any']:
, only 'none' is an accepted abi tag for pypy, but here we have pypy_41.

@kwlzn

This comment has been minimized.

Copy link
Member

kwlzn commented Mar 17, 2016

ah, makes sense. I think this is probably just a quick pypy-specific extension of #213 (which only covered cpython initially).

@kwlzn kwlzn changed the title issue with pypy Extend wheel ABI support for pypy. Mar 17, 2016

@kwlzn kwlzn added this to the improve pypy support milestone Aug 9, 2016

@Julian

This comment has been minimized.

Copy link

Julian commented Nov 9, 2016

+1

@kwlzn

This comment has been minimized.

Copy link
Member

kwlzn commented May 16, 2018

this should now be vastly improved as of pex 1.4.0 (which adds impl/version/abi targeting), but I don't have a concrete repro to prove that.

you may try kicking the tires with the new version, with tag targeting usage described here: #281 (comment)

I'm going to optimistically mark this as closed given much of the underlying resolver logic has changed, but feel free to reopen with a fresh repro if you're still having issues.

@kwlzn kwlzn closed this May 16, 2018

@mithrandi

This comment has been minimized.

Copy link

mithrandi commented May 16, 2018

This still doesn't quite work, seems like something weird with the platform detection:

# In a virtualenv here:
$ python -V
Python 2.7.13 (6.0.0+dfsg-1, Apr 27 2018, 22:07:01)
[PyPy 6.0.0 with GCC 7.3.0]
$ pex --version
pex 1.4.0
$ pex -vvv -c twistd -o twisted.pex Twisted
…
pex: Validated Twisted-18.4.0.tar.bz2 (sha256=a4cc164a781859c74de47f17f0e85f4bce8a3321a9d0892c015c8f80c4158ad9)
pex: Target package WheelPackage('file:///tmp/tmpgwzUzE/Twisted-18.4.0-pp260-pypy_41-linux_x86_64.whl') is not compatible with [('pp260', 'pp260pypy_41', 'manylinux1_x86_64'), ('pp260', 'pp260pypy_41', 'linux_x86_64'), ('pp260', 'none', 'manylinux1_x86_64'), ('pp260', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('pp260', 'none', 'any'), ('pp2', 'none', 'any'), ('py260', 'none', 'any'), ('py2', 'none', 'any')]
Traceback (most recent call last):
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/bin/pex.py", line 694, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/bin/pex.py", line 624, in build_pex
    for dist in resolveds:
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 525, in resolve_multi
    use_manylinux=use_manylinux):
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 451, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 248, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 321, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/mithrandi/deployment/virtualenvs/tempenv-054e1965378d4/site-packages/pex/resolver.py", line 206, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
Untranslateable: Package SourcePackage(u'file:///home/mithrandi/.pex/build/Twisted-18.4.0.tar.bz2') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

Seems like pp260 is landing up in ('pp260', 'pp260pypy_41', 'linux_x86_64') twice for some reason.

@mithrandi

This comment has been minimized.

Copy link

mithrandi commented May 16, 2018

Aha, the problem is this logic here:

pex/pex/platforms.py

Lines 50 to 55 in 2208cb0

@staticmethod
def _maybe_prefix_abi(impl, version, abi):
# N.B. This permits users to pass in simpler extended platform strings like
# `linux-x86_64-cp-27-mu` vs e.g. `linux-x86_64-cp-27-cp27mu`.
impl_ver = ''.join((impl, version))
return abi if abi.startswith(impl_ver) else ''.join((impl_ver, abi))

Since pypy_41 does not begin with pp260, it gets prefixed with it, which is wrong for PyPy ABIs. If I pass --platform linux_x86_64 explicitly then it works!

$ ./twisted.pex --version
twistd (the Twisted daemon) 18.4.0
Copyright (c) 2001-2018 Twisted Matrix Laboratories.
See LICENSE for details.
@kwlzn

This comment has been minimized.

Copy link
Member

kwlzn commented May 16, 2018

nice. I've cut #483 to repair the abi prefixing for pypy.

@kwlzn kwlzn reopened this May 16, 2018

@kwlzn kwlzn closed this in #483 May 16, 2018

kwlzn added a commit that referenced this issue May 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.