New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3 - fix cffi resolver issues #6225

Merged
merged 3 commits into from Jul 24, 2018

Conversation

Projects
None yet
2 participants
@Eric-Arellano
Copy link
Contributor

Eric-Arellano commented Jul 24, 2018

Problem

When running any unit test that extends TestBase with Python 3 on macOS (not tested on Linux), pants throws this exception

Exception message: Package SourcePackage(u'file:///Users/earellano/.cache/pants/python_cache/requirements/CPython-3.7.0/cryptography-2.3.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

I realized this is because I didn't have liffbfi and cffi installed, so I followed their install instructions. Our README already specifies you need these, so this is okay so far.

After installing liffbfi and cffi, I ran into a new exception:

Traceback (most recent call last):
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 157, in save_modules
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 198, in setup_context
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 248, in run_setup
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 278, in run
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 246, in runner
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 47, in _execfile
  File "/var/folders/3q/rl02b2k936bb2x8mqfg48t180000gn/T/easy_install-ijy2a6ow/pycparser-2.18/setup.py", line 54, in <module>

  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 209, in run
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 245, in zip_safe
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 355, in analyze_egg
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 392, in scan_module
ValueError: bad marshal data (unknown type code)

Solution

Bumping setuptools to the max possible, 33.1.1, fixes the issue for me, when followed by ./pants clean-all

Note that setuptools goes all the way to 40.0, but PEX requires < 34.

To test locally

Add compatibility='CPython>=3.5', to the :meta entry in pants_test/util/BUILD. Then run

./pants test tests/python/pants_test/util:meta

While the unit test itself will fail, the requirements resolver should pass.

@@ -41,7 +41,7 @@ def execute(self):
safe_concurrent_rename(pants_wd, tmp_trash)
safe_concurrent_rename(tmpdir, pants_wd)

if self.get_options().async:
if self.get_options()['async']:

This comment has been minimized.

@Eric-Arellano

Eric-Arellano Jul 24, 2018

Contributor

async is a reserved word in Py3

new_exception = self.CacheValidationError("Problem validating target {} in {}: {}"
.format(target.id, target.address.spec_path, e))

raise self.CacheValidationError, new_exception, exc_info[2]
raise_from(self.CacheValidationError(new_exception), e)

This comment has been minimized.

@Eric-Arellano

Eric-Arellano Jul 24, 2018

Contributor

raise from is Python 3 syntactic sugar that provides the same functionality as exc_info[2].

@@ -27,7 +27,7 @@ pywatchman==1.4.1
requests[security]>=2.5.0,<2.19
scandir==1.2
setproctitle==1.1.10
setuptools==30.0.0
setuptools==33.1.1

This comment has been minimized.

@Eric-Arellano

Eric-Arellano Jul 24, 2018

Contributor

Good catch. Fixed with latest commit. Thanks!

@jsirois jsirois merged commit d9316dd into pantsbuild:master Jul 24, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@Eric-Arellano Eric-Arellano deleted the Eric-Arellano:py3-fixes_cffi-issues branch Jul 25, 2018

CMLivingston pushed a commit to CMLivingston/pants that referenced this pull request Aug 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment