pants pyprep fails on setuptools

[dantswain]

When I run PEX_VERBOSE=5 ./pants -ldebug pyprep --pyprep-interpreter-fail-fast I get a failure

Exception message: Package SourcePackage(u'https://files.pythonhosted.org/packages/66/6d/dad0d39ce1cfa98ef3634463926e7324e342c956aecb066968e2e3696300/setuptools-30.0.0.tar.gz#sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91') is not translateable by ChainedTranslator(EggTranslator, SourceTranslator)

I think this must be a problem with my environment, but I'm having a really hard time tracking down what could be causing it. I'm on macos 10.13.6 and using pyenv to set python to 2.7.12. I'll try to provide relevant details below but I can provide more if it would be useful.

which python2.7
/Users/dswain/.pyenv/shims/python2.7

python --version
Python 2.7.12.  # from pyenv

echo $PATH
/Users/dswain/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/dswain/.pyenv/shims

cat pants.ini
[python-setup]

[python-repos]

[GLOBAL]
pants_version: 1.7.0

There's a lot of output from the command but here is where it seems to start going sideways:

09:38:25 00:01   [pyprep]
09:38:25 00:01     [interpreter]DEBUG] acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x19aa15a6850>
pex: Could not identify /Users/dswain/.pyenv/shims/python3.6: received exit code 127 during execution of `['/Users/dswain/.pyenv/shims/python3.6']` while trying to execute `['/Users/dswain/.pyenv/shims/python3.6']`
pex: Could not identify /Users/dswain/.pyenv/shims/python3.6: received exit code 127 during execution of `['/Users/dswain/.pyenv/shims/python3.6']` while trying to execute `['/Users/dswain/.pyenv/shims/python3.6']`
DEBUG] Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None)
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x19aa15c34d0>
pex: crawling link i=0 link=Link(u'https://pypi.python.org/simple/setuptools/') follow_links=False
DEBUG] Starting new HTTPS connection (1): pypi.python.org
DEBUG] https://pypi.python.org:443 "HEAD /simple/setuptools/ HTTP/1.1" 301 0
DEBUG] Starting new HTTPS connection (1): pypi.org
DEBUG] https://pypi.org:443 "HEAD /simple/setuptools/ HTTP/1.1" 200 0
DEBUG] https://pypi.org:443 "GET /simple/setuptools/ HTTP/1.1" 200 82462
pex: Fetching https://files.pythonhosted.org/packages/66/6d/dad0d39ce1cfa98ef3634463926e7324e342c956aecb066968e2e3696300/setuptools-30.0.0.tar.gz#sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91 :: Fetching https://files.DEBUG] Starting new HTTPS connection (1): files.pythonhosted.orge342c956aecb066968e2e3696300/setuptools-30.0.0.tar.gz#sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91
DEBUG] https://files.pythonhosted.org:443 "GET /packages/66/6d/dad0d39ce1cfa98ef3634463926e7324e342c956aecb066968e2e3696300/setuptools-30.0.0.tar.gz HTTP/1.1" 200 647961
pex: Validated setuptools-30.0.0.tar.gz (sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91)                                                                                                                                   
pex: Validated setuptools-30.0.0.tar.gz (sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91)
pex: Fetching https://files.pythonhosted.org/packages/66/6d/dad0d39ce1cfa98ef3634463926e7324e342c956aecb066968e2e3696300/setuptools-30.0.0.tar.gz#sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91: 1102.2ms
pex:   Fetching https://files.pythonhosted.org/packages/66/6d/dad0d39ce1cfa98ef3634463926e7324e342c956aecb066968e2e3696300/setuptools-30.0.0.tar.gz#sha256=4e288421e62739fa59d1340f108e1db7559e33cd1e9719cf6bad479d188a7f91: 1101.4ms
pex: Target package EggPackage('file:///private/var/folders/rx/0dxh09ts3kj5r7l5w3v45jxh0000gn/T/tmpEiKmBo/setuptools-30.0.0-py3.6.egg') is not compatible with CPython-2.7.12 / macosx-10.13-intel
pex: Translating /private/var/folders/rx/0dxh09ts3kj5r7l5w3v45jxh0000gn/T/tmpqucpGR/setuptools-30.0.0.tar.gz into distribution: 951.0ms
pex:   Packaging setuptools: 851.1ms
pex:     Installing /var/folders/rx/0dxh09ts3kj5r7l5w3v45jxh0000gn/T/tmpupznMo: 850.0ms
DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x19aa15a6850>
DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x104822510>

I find these parts to be especially weird but I could be barking up the wrong tree:

pex: Could not identify /Users/dswain/.pyenv/shims/python3.6: received exit code 127 during execution of `['/Users/dswain/.pyenv/shims/python3.6']` while trying to execute `['/Users/dswain/.pyenv/shims/python3.6']`
pex: Target package EggPackage('file:///private/var/folders/rx/0dxh09ts3kj5r7l5w3v45jxh0000gn/T/tmpEiKmBo/setuptools-30.0.0-py3.6.egg') is not compatible with CPython-2.7.12 / macosx-10.13-intel

How is python3.6 getting involved? Why is it trying to use the 3.6 egg of setuptools?

Note I've isolated this down to an empty directory with nothing other than the pants script and the pants.ini file I posted above.

[jsirois]

The python3 output is a red-herring. Pants bootstraps an interpreter cache with all known pythons before selecting down to one that meets your constraints and then populating that interpreter with setuptools and wheel.

If you up PEX_VERBOSE to 9 we'll see the indisputable proof, but even with your output there is:

pex: Target package EggPackage('file:///private/var/folders/rx/0dxh09ts3kj5r7l5w3v45jxh0000gn/T/tmpEiKmBo/setuptools-30.0.0-py3.6.egg') is not compatible with CPython-2.7.12 / macosx-10.13-intel

The macosx-10.13-intel is not a valid platform identifier. We had to work around this in Pants CI itself for the OSX system-provided 2.7.10 which also does this: a1696c8#diff-354f30a63fb0907d4ad57269548329e3R479

Your options are:

1. Don't use OSX ;) - it often provides broken bespoke interpreters that require workarounds.

2. Ensure the problematic python interpreter is never selected by modifying [python-setup] interpreter_constraints to bump the floor. By default it's set to:

    $ ./pants options --scope=python-setup --name=interpreter_constraints
    python-setup.interpreter_constraints = ['CPython>=2.7,<3'] (from HARDCODED)
   

3. Like 2, but use [python-setup] interpreter_search_paths. You can learn more from ./pants python-setup --help-advanced.

4. Chip in or wait for OSX's python 2.7.10 interpreter reports a bad local platform, bdists fail to resolve pex-tool/pex#523 to get fixed and be used in Pants.

[dantswain]

Thanks @jsirois

I tried setting the interpreter_constraints to CPython>2.7.10,<3 but that didn't help. I'm not understanding why it would still try to install the py3.6 egg.

Setting interpreter_search_paths to the bin directory of the version of python I'm using from pyenv (~/.pyenv/versions/2.7.12/bin) solves the problem for me but it's not a very portable solution. Is this controllable with an env var?

FWIW This worked fine on a different machine that is also on macos 10.13 and using pyenv. There's something about the env that's different but I can't figure out what that is.

[Eric-Arellano]

Sorry for the very long delay in replying!

Is this controllable with an env var?

Yes, PANTS_PYTHON_SETUP_INTERPRETER_SEARCH_PATHS. All options may be specified through a config file, env var, and/or command line arg. See https://www.pantsbuild.org/docs/options.

FYI we recently released Pants 2.0, which overhauls Pants from the ground-up to have more robust/polished Python support, better caching, more concurrency, and a dynamic UI. Check out https://blog.pantsbuild.org/introducing-pants-v2/.