Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add first-class support for multi-interpreter and multi-platform pex construction. #394

Merged
merged 3 commits into from Jul 25, 2017

Conversation

@kwlzn
Copy link
Member

@kwlzn kwlzn commented Jul 24, 2017

This revives and extends the prior attempt in #201 to forklift the multi-interpreter/platform resolve behavior from pants directly into pex to better support general usage via pex(1) (and transitively, the Heron team's Bazel+pex integration).

Fixes #68.

@kwlzn kwlzn force-pushed the kwlzn:kwlzn/multiplatform_pex branch 6 times, most recently from beb3132 to 85b8a75 Jul 25, 2017
@kwlzn kwlzn force-pushed the kwlzn:kwlzn/multiplatform_pex branch from 85b8a75 to c8b5bdf Jul 25, 2017
@kwlzn kwlzn requested review from jsirois and stuhood Jul 25, 2017
@@ -290,8 +293,11 @@ def configure_clp_pex_environment(parser):
group.add_option(
'--platform',
dest='platform',
default=Platform.current(),
help='The platform for which to build the PEX. Default: %default')
default=[],

This comment has been minimized.

@jsirois

jsirois Jul 25, 2017
Member

How about [Platform.current()] here to make the help clearer (%default)? I'd be game for the duplicate [Platform.current()] defaulting code in resolve_multi if extracting a default_platforms function to some reasonable location for both to use seems onerous / more trouble than it's worth.

This comment has been minimized.

@kwlzn

kwlzn Jul 25, 2017
Author Member

ah, how would you feel about just replacing the %default here with a literal description?

the problem with defaulting to [Platform.current()] here is that the action=append mode is additive only, so without implementing a custom callback action for this option you'd end up resolving/building for the local platform every time whether or not you actually intended to.

This comment has been minimized.

@jsirois

jsirois Jul 25, 2017
Member

Makes sense, sgtm

This comment has been minimized.

@kwlzn

kwlzn Jul 25, 2017
Author Member

fixed.

with TRACER.timed('Resolving interpreter', V=2):
interpreter = interpreter_from_options(options)
def _lowest_version_interpreter(interpreters):
"""Given an iterable of interpreters, return the one with the lowest version."""

This comment has been minimized.

@jsirois

jsirois Jul 25, 2017
Member

More than an iterable. Just say list?

This comment has been minimized.

@kwlzn

kwlzn Jul 25, 2017
Author Member

fixed.

"""Given an iterable of interpreters, return the one with the lowest version."""
lowest = interpreters[0]
for i in interpreters[1:]:
lowest = lowest if lowest < i else i

This comment has been minimized.

@jsirois

jsirois Jul 25, 2017
Member

This implies the interpreter objects implement < which suggests sorted(interpreters)[0] (or -1, I'm never sure!). If this thinking is correct, then the comment above is moot.

This comment has been minimized.

@kwlzn

kwlzn Jul 25, 2017
Author Member

I considered that and attempted it, but given that PythonInterpreter only implements __lt__ I was worried that sorted could misbehave - so stuck to use of the bare < operator only here.

I tried to add functools.@total_ordering to PythonInterpreter, which should've made sorted 100% safe, but that failed in the 2.6 shard due to not being present - so stuck with a 2.6-safe approach for the time being.

This comment has been minimized.

@jsirois

jsirois Jul 25, 2017
Member

Gotcha, makes sense.

@kwlzn kwlzn merged commit 862d80a into pantsbuild:master Jul 25, 2017
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
kwlzn added a commit that referenced this pull request Aug 25, 2017
This PR adds a new CLI option, -p or --preamble-file, that is used to provide a preamble to the pex builder object.

We use pex via its API in the internal build tooling at NerdWallet (👋, we're neighbors!). We've been leveraging it for multi-platform support, but now that #394 / 1.2.9 has landed the only thing missing from the CLI that we leverage is the preamble in the builder, which was pretty easy to hook up. Once this PR has landed we can remove all of the API integration and just rely on the pex CLI!

The CLI option and the PEXBuilder preamble functionality are covered by tests.

Local functional testing:

(pex-foo) 
evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:49:54 08.16)
#536 ❯❯❯ echo 'print "foo!"' > preamble
(pex-foo) 
evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:50:11 08.16)
#537 ❯❯❯ pex -p preamble 
foo!
Python 2.7.13 (default, Jun  7 2017, 11:02:53) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
(pex-foo) 
evanborgstrom@evanborgstrom /tmp/preamble — u:34 j:1 (21:50:18 08.16)
#538 ❯❯❯ pex 
Python 2.7.13 (default, Jun  7 2017, 11:02:53) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants
You can’t perform that action at this time.