New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'current' platform handling. #6104

Merged
merged 5 commits into from Jul 13, 2018

Conversation

Projects
None yet
6 participants
@jsirois
Copy link
Member

jsirois commented Jul 12, 2018

Previously the platform passed for 'current' was partial, leading to
ambiguous resolves. Further, partial platforms in user's BUILD files
also would lead to ambiguous resolves. For example, a multi-platform
python_binary with something like
platforms=['current', 'linux-x86_64', 'macosx_10.11_x86_64'].

Expand and and then fixup platforms as required by replacing
get_local_platform with expand_and_maybe_adjust_platform.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 12, 2018

5 Legit looking errors in tests/python/pants_test/backend/python/tasks/test_python_distribution_integration.py, all 5 of the form:

...
                     E   	16:55:41 00:07   [binary]
                     E   	16:55:41 00:07     [binary-jvm-prep-command]
                     E   	16:55:41 00:07       [jvm_prep_command]
                     E   	16:55:41 00:07     [binary-prep-command]
                     E   	16:55:41 00:07     [py]
                     E   	                   Invalidated 1 target.
                     E   	                   created pex ../../../../../var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpUEmFhK/main.pex
                     E   	16:55:42 00:08     [py-wheels]
                     E   	                   created wheel ../../../../../var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpUEmFhK/fasthello-1.0.0-cp27-cp27m-macosx_10_12_x86_64.whl
                     E   	16:55:42 00:08     [jvm]
                     E   	16:55:42 00:08     [dup]
                     E   	16:55:42 00:08     [cpplib]
                     E   	16:55:42 00:08       [cpp-library]
                     E   	16:55:42 00:08     [cpp]
                     E   	16:55:42 00:08       [cpp-binary]
                     E   	16:55:42 00:08     [go]
                     E   	16:55:42 00:08   [run]
                     E   	16:55:42 00:08     [py]
                     E   	                   Invalidated 3 targets.
                     E   	16:55:42 00:08       [run]
                     E   	
                     E   	FAILURE: /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 example.python_distribution.hello.fasthello.main  ... exited non-zero (1)
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	16:55:43 00:09   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for:
                     E   	fasthello
                      PythonDistributionIntegrationTest.test_pants_run 
...
@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 12, 2018

A similar issue in tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py:

tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py::PytestRunIntegrationTest::test_pants_test_interpreter_selection_with_pexrc <- pyprep/sources/95e7d418de90040eb18b826381cad395b8bf7450/pants_test/backend/python/tasks/test_pytest_run_integration.py
...
                     E   	17:15:57 00:07   [test]
                     E   	17:15:57 00:07     [test-jvm-prep-command]
                     E   	17:15:57 00:07       [jvm_prep_command]
                     E   	17:15:57 00:07     [test-prep-command]
                     E   	17:15:57 00:07     [test]
                     E   	17:15:57 00:07     [pytest-prep]
                     E   	                   Invalidated 3 targets.
                     E   	17:16:00 00:10     [pytest]
                     E   	                   Invalidated 1 target.
                     E   	                   scrubbed PYTHONPATH=/home/travis/build/pantsbuild/pants/src/python: from py.test environment
                     E   	17:16:00 00:10       [run]
                     E   	                     Failed to execute PEX file, missing linux_x86_64-cp-34-cp34m compatible dependencies for:
                     E   	                     coverage
                     E   	                     more-itertools
                     E   	                     pytest
                     E   	                     pytest-cov
                     E   	                     
                     E   	                   testprojects/src/python/interpreter_selection/python_3_selection_testing:test_py3.....   SUCCESSm
                     E   	FAILURE
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	17:16:01 00:11   [complete]
                     E   	               FAILURE
...

And a slightly different looking error as well:

tests/python/pants_test/backend/python/tasks/test_python_run_integration.py::PythonRunIntegrationTest::test_pex_resolver_blacklist_integration <- pyprep/sources/95e7d418de90040eb18b826381cad395b8bf7450/pants_test/backend/python/tasks/test_python_run_integration.py
...
                     E   	17:16:13 00:01   [pyprep]
                     E   	17:16:13 00:01     [interpreter]
                     E   	                   Invalidated 1 target.
                     E   	17:16:15 00:03     [build-local-dists]
                     E   	17:16:15 00:03     [requirements]
                     E   	                   Invalidated 1 target.
                     E   	               Waiting for background workers to finish.
                     E   	17:16:20 00:08   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	INFO] For async removal, run `./pants clean-all --async`
                     E   	
                     E   	Exception caught: (<class 'pex.resolver.Untranslateable'>)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 72, in <module>
                     E   	    main()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 68, in main
                     E   	    PantsLoader.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 64, in run
                     E   	    cls.load_and_execute(entrypoint)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 57, in load_and_execute
                     E   	    entrypoint_main()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_exe.py", line 38, in main
                     E   	    PantsRunner(exiter, start_time=start_time).run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_runner.py", line 52, in run
                     E   	    return runner.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 45, in run
                     E   	    self._run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 93, in _run
                     E   	    goal_runner_result = goal_runner.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/goal_runner.py", line 221, in run
                     E   	    result = self._execute_engine()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/goal_runner.py", line 209, in _execute_engine
                     E   	    result = engine.execute(self._context, self._goals)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/legacy_engine.py", line 25, in execute
                     E   	    self.attempt(context, goals)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/round_engine.py", line 232, in attempt
                     E   	    goal_executor.attempt(explain)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/round_engine.py", line 48, in attempt
                     E   	    task.execute()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/resolve_requirements.py", line 29, in execute
                     E   	    pex = self.resolve_requirements(interpreter, self.context.targets(has_python_requirements))
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/resolve_requirements_task_base.py", line 80, in resolve_requirements
                     E   	    dump_requirement_libs(builder, interpreter, req_libs, self.context.log, platforms=maybe_platforms)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 95, in dump_requirement_libs
                     E   	    dump_requirements(builder, interpreter, reqs, log, platforms=platforms)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 119, in dump_requirements
                     E   	    distributions = resolve_multi(interpreter, deduped_reqs, platforms, find_links)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 163, in resolve_multi
                     E   	    use_manylinux=python_setup.use_manylinux)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 453, in resolve
                     E   	    return resolver.resolve(resolvables_from_iterable(requirements, builder))
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 248, in resolve
                     E   	    dist = self.build(package, resolvable.options)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 323, in build
                     E   	    dist = super(CachingResolver, self).build(package, options)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 206, in build
                     E   	    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
                     E   	
                     E   	Exception message: Package SourcePackage(u'file:///home/travis/.cache/pants/python_cache/requirements/CPython-3.4.3/pandocfilters-1.4.2.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)m
...
@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

Eric-Arellano commented Jul 12, 2018

@jsirois I've been getting that first error ever since I started working on this project two weeks ago when running ci.sh on my Mac.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 12, 2018

Thanks for the independent datapoint @Eric-Arellano. I run linux so this is especially useful to me.

That says we definitely have more problems than just in resolve-multi which is the only bit this PR fixes.

@CMLivingston

This comment has been minimized.

Copy link
Contributor

CMLivingston commented Jul 12, 2018

@jsirois I have a good recollection of that platform-based python_distribution test and the case it is covering. I can look into that.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 12, 2018

Thanks @CMLivingston

@kwlzn

kwlzn approved these changes Jul 12, 2018

@CMLivingston

This comment has been minimized.

Copy link
Contributor

CMLivingston commented Jul 12, 2018

I cannot repro the osx shard failure locally (https://travis-ci.org/pantsbuild/pants/jobs/403189175). Would anyone else be able to take a crack at the tests and report any failures?
./pants test tests/python/pants_test/backend/python/tasks:: -- -k test_python_distribution_integration

From what I can see in the output log, it looks like the wheel is named properly and the os version aligns with mine, so I can't see why this would fail.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

OK, This hacky fix debug output (see: >>> Modifying given platform of 'current':) should also summarize the problem for at least the test_pants_test_interpreter_selection_with_pexrc error above:

                    18:50:55 00:11     [pytest-prep]
                                        Invalidated 3 targets.
                             >>> Modifying given platform of 'current':
                                 Resolves as Platform(platform='linux_x86_64', impl='cp', version='27', abi='cp27mu')
                                 Under current interpreter u"PythonInterpreter('/home/jsirois/dev/pantsbuild/jsirois-pants/build-support/pants_dev_deps.venv/bin/python2.7', PythonIdentity('CPython', 2, 7, 15))"
                                 
                                 To match given interpreter u"PythonInterpreter('/usr/bin/python3.6', PythonIdentity('CPython', 3, 6, 6))".
                                 
                                 Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='36', abi='cp36m')
                     
                     18:51:06 00:22     [pytest]
                                        Invalidated 1 target.
                                        scrubbed PYTHONPATH=/home/jsirois/dev/pantsbuild/jsirois-pants/src/python: from py.test environment
                     18:51:06 00:22       [run]
                                          ============================= test session starts ==============================
                                          platform linux -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
                                          rootdir: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d, inifile: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d/test/pytest-prep/CPython-3.6.6/ef25b5846bb2b47d95d4248cab9384a52f73af8b/pytest.ini
                                          plugins: timeout-1.2.1, cov-2.4.0
                                          collected 1 item
                                          
                                          testprojects/src/python/interpreter_selection/python_3_selection_testing/test_py3.py . [100%]
                                          
                                           generated xml file: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d/test/pytest/testprojects.src.python.interpreter_selection.python_3_selection_testing.test_py3/junitxml/TEST-testprojects.src.python.interpreter_selection.python_3_selection_testing.test_py3.xml 
                                          =========================== 1 passed in 0.04 seconds ===========================
                                          
                                        testprojects/src/python/interpreter_selection/python_3_selection_testing:test_py3.....   SUCCESS
@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

This is more work related to workarounds for pantsbuild/pex#511. Most of the logic introduced here - or all of it - should move into pex.resolver.resolve.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

OK @CMLivingston all linux test failures are now fixed so I'll dig into test_python_distribution_integration failing on OSX.

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

Noting that in the OSX failures, ./pants, via which and the PATH selects /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/bin/python2.7 as the runtime interpreter and the test failures all show /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 - selected no doubt by the min here. So two different local interpreters are in-play.

The fix above is designed to handle that case in the resolve_multi path. I suspect the python distribution code represents another - unfixed path.

@@ -227,7 +227,8 @@ def _create_dist(self, dist_tgt, dist_target_dir, interpreter,

setup_runner = SetupPyRunner.for_bdist_wheel(
dist_target_dir,
is_platform_specific=is_platform_specific)
is_platform_specific=is_platform_specific,
interpreter=interpreter)

This comment has been minimized.

@jsirois

jsirois Jul 13, 2018

Member

OK @CMLivingston and @cosmicexplorer - other issues elsewhere aside - this code never stood a fair chance of working since the selected interpreter was not passed.

@cosmicexplorer

This comment has been minimized.

Copy link
Contributor

cosmicexplorer commented Jul 13, 2018

The interpreter selection bit you just described seems utterly fantastic because

stderr:
                     E   	Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for:
                     E   	fasthello

is what i spent an entire migraine on trying to understand in the osx shard in #6022 last week (which you and Kris were there for, which I appreciate immensely) — not trying to whine, just this seems like it would directly address that maybe, so that’s great.

@jsirois jsirois force-pushed the jsirois:python/resolve/fix-current-platform-handling branch from 9bd378a to ab976db Jul 13, 2018

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

Noting current issue on OSX:

With the correct interpreter being used for run/test/repl/... pexes, which is the system 2.7.10 in this case (as opposed to brew 2.7.14), the accepted platforms calculation looks wrong - no macosx_10_*_x86_64, just macosx_10_*_intel:

pex: E: tags for 'macosx-10.12-intel' x PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', PythonIdentity('CPython', 2, 7, 10), {('matplotlib', '1.3.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-corelocation', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-coretext', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-syncservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('six', '1.4.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-fsevents', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-cocoa', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-preferencepanes', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('zope.interface', '4.1.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyparsing', '2.0.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-collaboration', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pytz', '2013.7'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-pubsub', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-scriptingbridge', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-servicemanagement', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('numpy', '1.8.0rc1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-installerplugins', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('setuptools', '18.5'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-opendirectory', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-applescriptkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-automator', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-addressbook', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-searchkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-applescriptobjc', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-qtkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-latentsemanticmapping', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('scipy', '0.13.0b1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('modulegraph', '0.10.4'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('altgraph', '0.10.2'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-instantmessage', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('xattr', '0.6.4'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-eventkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-social', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-cfnetwork', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-quartz', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-exceptionhandling', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-launchservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-coredata', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('python', '2.7.10'): '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', ('bdist-mpkg', '0.5.0'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-message', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-inputmethodkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('py2app', '0.7.3'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-dictionaryservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-accounts', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-screensaver', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyopenssl', '0.13.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('bonjour-py', '0.3'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-core', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('wsgiref', '0.1.2'): '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', ('pyobjc-framework-webkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('python-dateutil', '1.5'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-systemconfiguration', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('macholib', '1.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python'}) -> [('cp27', 'cp27m', 'macosx_10_8_universal'), ('cp27', 'cp27m', 'macosx_10_12_universal'), ('cp27', 'cp27mu', 'macosx_10_7_intel'), ('cp27', 'cp27mu', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_9_intel'), ('cp27', 'cp27d', 'macosx_10_12_universal'), ('cp27', 'cp27dm', 'macosx_10_8_intel'), ('cp27', 'cp27dmu', 'macosx_10_0_universal'), ('cp27', 'cp27d', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_10_intel'), ('cp27', 'cp27dm', 'macosx_10_8_universal'), ('cp27', 'cp27dm', 'macosx_10_11_universal'), ('cp27', 'cp27m', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_9_intel'), ('cp27', 'cp27dm', 'macosx_10_0_universal'), ('py2', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_12_intel'), ('cp27', 'cp27m', 'macosx_10_5_universal'), ('cp2', 'none', 'any'), ('cp27', 'cp27m', 'macosx_10_8_intel'), ('py2', 'none', 'macosx_10_11_universal'), ('cp27', 'cp27m', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_10_intel'), ('py2', 'none', 'any'), ('cp27', 'none', 'macosx_10_5_intel'), ('cp27', 'none', 'macosx_10_8_intel'), ('cp27', 'cp27mu', 'macosx_10_0_universal'), ('cp27', 'cp27u', 'macosx_10_9_universal'), ('cp27', 'cp27mu', 'macosx_10_11_universal'), ('cp27', 'none', 'macosx_10_12_intel'), ('py27', 'none', 'any'), ('cp27', 'cp27dm', 'macosx_10_4_intel'), ('cp27', 'cp27mu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_6_intel'), ('cp27', 'cp27mu', 'macosx_10_9_universal'), ('cp27', 'cp27mu', 'macosx_10_5_intel'), ('cp27', 'cp27dmu', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_3_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_intel'), ('cp27', 'cp27dm', 'macosx_10_7_intel'), ('cp27', 'none', 'macosx_10_12_universal'), ('cp27', 'cp27u', 'macosx_10_4_intel'), ('cp27', 'none', 'macosx_10_11_intel'), ('cp27', 'cp27u', 'macosx_10_0_universal'), ('cp27', 'cp27m', 'macosx_10_12_intel'), ('cp27', 'none', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_11_universal'), ('cp27', 'cp27dmu', 'macosx_10_3_universal'), ('cp27', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_12_universal'), ('py2', 'none', 'macosx_10_9_universal'), ('cp27', 'cp27dm', 'macosx_10_9_universal'), ('cp27', 'cp27u', 'macosx_10_11_universal'), ('cp27', 'cp27dm', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_8_intel'), ('cp27', 'cp27u', 'macosx_10_1_universal'), ('cp27', 'none', 'macosx_10_10_universal'), ('cp27', 'cp27d', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_10_universal'), ('cp27', 'cp27mu', 'macosx_10_7_universal'), ('cp27', 'cp27mu', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27u', 'macosx_10_7_intel'), ('cp27', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_4_intel'), ('cp27', 'cp27dm', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_9_intel'), ('cp27', 'cp27m', 'macosx_10_7_intel'), ('cp27', 'cp27m', 'macosx_10_3_universal'), ('cp27', 'cp27m', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_11_intel'), ('cp27', 'cp27m', 'macosx_10_6_intel'), ('cp27', 'cp27dmu', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_8_intel'), ('cp27', 'cp27d', 'macosx_10_11_universal'), ('cp27', 'cp27mu', 'macosx_10_12_intel'), ('cp27', 'cp27u', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_8_universal'), ('cp27', 'cp27dm', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_9_universal'), ('cp27', 'cp27d', 'macosx_10_6_intel'), ('cp27', 'cp27mu', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_7_universal'), ('cp27', 'cp27dmu', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_12_intel'), ('cp27', 'cp27dmu', 'macosx_10_9_universal'), ('py2', 'none', 'macosx_10_8_universal'), ('py2', 'none', 'macosx_10_12_universal'), ('cp27', 'cp27mu', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_10_intel'), ('cp27', 'cp27mu', 'macosx_10_4_universal'), ('cp27', 'cp27dm', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_10_universal'), ('cp27', 'cp27u', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_8_universal'), ('cp27', 'cp27mu', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_7_intel'), ('cp27', 'cp27dm', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_5_intel'), ('cp27', 'cp27d', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_intel'), ('cp27', 'cp27u', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_6_universal'), ('cp27', 'cp27mu', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_9_universal'), ('cp27', 'cp27dm', 'macosx_10_10_universal'), ('cp27', 'cp27dmu', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_8_intel'), ('cp27', 'cp27u', 'macosx_10_10_intel'), ('cp27', 'cp27dmu', 'macosx_10_9_intel'), ('cp27', 'none', 'macosx_10_8_universal'), ('py2', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_12_intel'), ('cp27', 'cp27d', 'macosx_10_0_universal'), ('cp27', 'none', 'any'), ('cp27', 'cp27m', 'macosx_10_9_universal'), ('cp27', 'cp27d', 'macosx_10_7_universal'), ('cp27', 'none', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_8_intel'), ('cp27', 'cp27dm', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_11_universal'), ('cp27', 'cp27u', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_10_intel'), ('cp27', 'cp27dmu', 'macosx_10_8_universal'), ('cp27', 'cp27d', 'macosx_10_12_intel'), ('cp27', 'cp27u', 'macosx_10_12_intel'), ('cp27', 'none', 'macosx_10_7_intel'), ('cp27', 'cp27u', 'macosx_10_6_intel'), ('py2', 'none', 'macosx_10_3_universal'), ('py2', 'none', 'macosx_10_5_universal'), ('py2', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27dmu', 'macosx_10_7_intel'), ('cp27', 'cp27d', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_3_universal'), ('py2', 'none', 'macosx_10_1_universal'), ('py2', 'none', 'macosx_10_10_universal'), ('cp27', 'cp27dm', 'macosx_10_5_intel'), ('cp27', 'cp27d', 'macosx_10_5_universal'), ('cp27', 'cp27u', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_11_intel'), ('cp27', 'cp27dm', 'macosx_10_7_universal'), ('cp27', 'cp27u', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_7_intel'), ('py2', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_8_universal'), ('cp27', 'cp27u', 'macosx_10_8_intel'), ('cp27', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_11_intel'), ('cp27', 'none', 'macosx_10_11_universal'), ('cp27', 'none', 'macosx_10_7_universal'), ('cp27', 'cp27dm', 'macosx_10_3_universal')]

@jsirois jsirois force-pushed the jsirois:python/resolve/fix-current-platform-handling branch 2 times, most recently from 6196ae2 to 503b193 Jul 13, 2018

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

Eric-Arellano commented Jul 13, 2018

Awesome! I'm running ci.sh on mac - will let you know what happens.

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

Eric-Arellano commented Jul 13, 2018

6 tests fail now instead of I think it was 11 earlier, that's exciting! https://hastebin.com/oxinuwepol.rb

One of the failures is the same as CI's unit test failure, TestBuildLocalPythonDistributions.test_python_create_platform_specific_distribution

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

@Eric-Arellano note well that the OSX fix for Travis was ultimately getting off an OSX VM with Apple's supplied 2.7.10 which returns a broken -intel build platform. If you too have that, this may not provide you relief locally yet.

jsirois added some commits Jul 12, 2018

Fix 'current' platform handling.
Previously the platform passed for 'current' was partial, leading to
ambiguous resolves. Further, partial platforms in user's BUILD files
also would lead to ambiguous resolves. For example, a multi-platform
`python_binary` with something like
`platforms=['current', 'linux-x86_64', 'macosx_10.11_x86_64']`.

Expand and and then fixup platforms as required by replacing
`get_local_platform` with `expand_and_maybe_adjust_platform`.
Some `build_local_python_distributions` fixes.
+ Fix an unused `interpreter` parameter - forward to `SetupPyRunner`.
+ Kill declaring `--universal` on behalf of the package author. There
  is not enough info to make the decision if the code being dist'd, if
  pure python, is 2/3 compatible.

To compensate for the removal of `--universal`, give an example of how
to declare your python_dist as `--universal` when you know it is as the
BUILD `python_dist` target author.

@jsirois jsirois force-pushed the jsirois:python/resolve/fix-current-platform-handling branch from 833d40b to fe2907e Jul 13, 2018

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Jul 13, 2018

Thanks @Eric-Arellano - I was able to repro that failure on linux and it should be fixed this round.

@kwlzn

kwlzn approved these changes Jul 13, 2018

Copy link
Member

kwlzn left a comment

lgtm!

abi_tag=ident.abi_tag))


def expand_and_maybe_adjust_platform(interpreter, platform):

This comment has been minimized.

@kwlzn

kwlzn Jul 13, 2018

Member

a TODO to upstream this into pex might be good?

This comment has been minimized.

@jsirois

jsirois Jul 13, 2018

Member

Yup - this old one pantsbuild/pex#511, I'll add a TODO similar to the method above.

This comment has been minimized.

@jsirois

jsirois Jul 13, 2018

Member

Fixed.

env = copy(kwargs.pop('env', {}))

# Hack around bug in PEX where custom interpreters are not forwarded to PEXEnvironments.
# TODO(John Sirois): Remove when XXX is fixed.

This comment has been minimized.

@kwlzn

kwlzn Jul 13, 2018

Member

s/XXX/a ticket?

This comment has been minimized.

@jsirois

jsirois Jul 13, 2018

Member

Yup - just finished filing on the pex side. I'll send up a final comment-only diff with issue inks here as soon as this goes green then merge: pantsbuild/pex#522

This comment has been minimized.

@jsirois

jsirois Jul 13, 2018

Member

Fixed.

.travis.yml Outdated
@@ -476,6 +476,10 @@ matrix:
language: generic
env:
- SHARD="Rust + Platform-specific Tests OSX"
# Specifically avoid the OSX provided 2.7.10 under xcode8.3 since it returns a platform
# of `macosx-*-intel` where the `intel` suffix is bogus but pex has not yet been taught to
# deal with this. Can be removed when this issue is resolved: YYY

This comment has been minimized.

@jsirois

This comment has been minimized.

@jsirois

jsirois Jul 13, 2018

Member

Fixed.

@jsirois jsirois merged commit a1696c8 into pantsbuild:master Jul 13, 2018

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@jsirois jsirois deleted the jsirois:python/resolve/fix-current-platform-handling branch Jul 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment