Allow users to request a fixed python interpreter #4930

copumpkin · 2017-10-02T19:12:11Z

Problem

In some situations, the automatic python picker logic picks the wrong version of Python and we don't have much control over that beyond being able to specify version constraints (which sometimes aren't sufficient).

Solution

I added an option to explicitly specify a list of paths to search.

Result

Users can now pass in --python-setup-interpreter-search-paths=<search-paths> and have Pants search in the specified paths. You can even pass in the full path of the interpreter (e.g., which python) and it'll figure that out!

copumpkin · 2017-10-02T19:13:52Z

src/python/pants/backend/python/tasks2/select_interpreter.py

+    if fixed_interpreter:
+      interpreter = PythonInterpreter.from_binary(fixed_interpreter)
+    else:
+      python_tgts = self.context.targets(lambda tgt: isinstance(tgt, PythonTarget))


Use GitHub's secret whitespace ignore option to see that I haven't actually changed any of this: https://github.com/pantsbuild/pants/pull/4930/files?w=1

Neat! Didn't know that existed. Although I'm not sure what that does exactly, esp. where whitespace is meaningful.

Anyway, might be better to split this branch off into a separate method, for readability.

copumpkin · 2017-10-02T19:31:14Z

I'm torn between calling the option fixed vs. fixed-interpreter. The fixed version reads more nicely on the command line, so you can write --pyprep-interpreter-fixed=<interp>, but in pants.ini, the fixed-interpreter seems a bit clearer.

benjyw

Thanks for the contribution! See comments inline.

benjyw · 2017-10-02T20:24:03Z

src/python/pants/backend/python/tasks2/select_interpreter.py

@@ -41,6 +41,11 @@ class SelectInterpreter(Task):
  """Select an Python interpreter that matches the constraints of all targets in the working set."""

  @classmethod
+  def register_options(cls, register):
+    super(SelectInterpreter, cls).register_options(register)
+    register('--fixed-interpreter', type=str, help='Use this Python interpreter instead of searching based on constraints', default=None)


Maybe this option should be in python_setup.py along so it's next to --interpreter-constraints ?

Also, if I understand correctly the value here should be a path to a python binary? Let's update the help text to clarify this (i.e., what "this Python interpreter" means...)

Sure! I thought of doing that but then I don't know how to get the constraints option from there to here. I haven't figured out yet how the current top-level constraints propagate yet.

Ah, I need to use the global_instance I think. Okay, I'll push an update doing all that!

benjyw · 2017-10-02T20:26:27Z

src/python/pants/backend/python/tasks2/select_interpreter.py

+    if fixed_interpreter:
+      interpreter = PythonInterpreter.from_binary(fixed_interpreter)
+    else:
+      python_tgts = self.context.targets(lambda tgt: isinstance(tgt, PythonTarget))


Neat! Didn't know that existed. Although I'm not sure what that does exactly, esp. where whitespace is meaningful.

Anyway, might be better to split this branch off into a separate method, for readability.

copumpkin · 2017-10-02T21:15:39Z

@benjyw I pushed a new version that (I think) addresses your comments. I haven't run the tests or anything on it yet, but it seems to work fine on my local installation

kwlzn · 2017-10-02T21:19:08Z

src/python/pants/backend/python/subsystems/python_setup.py

@@ -62,13 +62,19 @@ def register_options(cls, register):
    register('--artifact-cache-dir', advanced=True, default=None, metavar='<dir>',
             help='The parent directory for the python artifact cache. '
                  'If unspecified, a standard path under the workdir is used.')
+    register('--fixed-interpreter', advanced=True, default=None, metavar='<interpreter-binary>',


this should probably be an interpreter search path that's checked against the constraints vs a single interpreter.. in order to support e.g. 2 + 3. we've learned this the hard way with PEX_PYTHON (vs e.g. PEX_PYTHONS).

So I'd pass those paths into interpreter_cache.setup()? I can do that

copumpkin · 2017-10-02T21:46:04Z

@kwlzn how's this one?

benjyw · 2017-10-03T03:59:49Z

src/python/pants/backend/python/subsystems/python_setup.py

@@ -62,13 +62,19 @@ def register_options(cls, register):
    register('--artifact-cache-dir', advanced=True, default=None, metavar='<dir>',
             help='The parent directory for the python artifact cache. '
                  'If unspecified, a standard path under the workdir is used.')
+    register('--interpreter-search-paths', advanced=True, type=list, default=[], metavar='<binary-paths>',
+             help='A list of paths in which to search for python')


"To search for python interpreters."

Tiny nit: End the help sentence with a period. For grammar and for consistency...

@benjyw fixed 😄

By default it looks in PATH (the old behavior) but you can now tell it to look elsewhere

copumpkin · 2017-10-05T13:53:45Z

@kwlzn @benjyw anything else I should change? I've been using this change locally on my 1.3 version and it seems to work fine

kwlzn

this LGTM - how about a quick integration test or two to keep it from regressing later tho?

copumpkin · 2017-10-05T21:59:39Z

@kwlzn I wasn't sure how best to test it, since I can't really assume multiple python versions will be present on the test box (or can I?) so it's hard for me to throw together a sample project that would make sure the resolution logic uses the python I want. I could set it to an empty search path and ensure that it complains that it can't find any pythons, but that feels kinda weak.

kwlzn · 2017-10-05T22:18:37Z

@copumpkin fair enough - tho there are some tests that target 2.7 + 3.x so I imagine it'd be possible.. but perhaps a couple too many hoops to jump through. can circle back after #4859 affords us better control over the test execution environment.

copumpkin · 2017-10-06T01:28:45Z

Cool, I'll keep an eye on #4859 then. Thanks again!

copumpkin · 2017-10-06T01:47:18Z

Oh also, should I backport this to the 1.3 branch? That's what I've been using it on in my team, but unsure if you want to put it into an official release.

copumpkin · 2017-10-09T17:23:57Z

Thanks @stuhood !

copumpkin · 2017-10-17T17:07:57Z

@kwlzn @benjyw @stuhood one thing I'm not sure about is how this should interact with the .pants.d cache of the python interpreter. If I change pants.ini to look for a different python version, I assume the cached python path will get updated, but I don't understand the cache more broadly to figure out how it should know about this option getting passed in. Any of you have any advice?

Problem In some situations, the automatic python picker logic picks the wrong version of Python and we don't have much control over that beyond being able to specify version constraints (which sometimes aren't sufficient). Solution I added an option to explicitly specify a list of paths to search. Result Users can now pass in --python-setup-interpreter-search-paths=<search-paths> and have Pants search in the specified paths. You can even pass in the full path of the interpreter (e.g., which python) and it'll figure that out!

copumpkin commented Oct 2, 2017

View reviewed changes

stuhood requested review from benjyw and kwlzn October 2, 2017 19:20

copumpkin force-pushed the fixed-python branch from 46bbe64 to 8689dda Compare October 2, 2017 19:23

copumpkin changed the base branch from 1.3.x to master October 2, 2017 19:23

benjyw reviewed Oct 2, 2017

View reviewed changes

copumpkin force-pushed the fixed-python branch from 8689dda to 6768dad Compare October 2, 2017 21:14

kwlzn requested changes Oct 2, 2017

View reviewed changes

copumpkin force-pushed the fixed-python branch from 6768dad to 6a4082b Compare October 2, 2017 21:45

benjyw reviewed Oct 3, 2017

View reviewed changes

Allow users to tell pants where to look for python interpreters

17d65ee

By default it looks in PATH (the old behavior) but you can now tell it to look elsewhere

copumpkin force-pushed the fixed-python branch from 6a4082b to 17d65ee Compare October 3, 2017 05:12

kwlzn approved these changes Oct 5, 2017

View reviewed changes

benjyw approved these changes Oct 5, 2017

View reviewed changes

kwlzn merged commit 6bc4ce9 into pantsbuild:master Oct 5, 2017

stuhood added the needs-cherrypick label Oct 9, 2017

stuhood added this to the 1.3.1 milestone Oct 9, 2017

stuhood removed the needs-cherrypick label Nov 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow users to request a fixed python interpreter #4930

Allow users to request a fixed python interpreter #4930

copumpkin commented Oct 2, 2017 •

edited

copumpkin Oct 2, 2017

benjyw Oct 2, 2017

copumpkin commented Oct 2, 2017

benjyw left a comment

benjyw Oct 2, 2017

copumpkin Oct 2, 2017

copumpkin Oct 2, 2017

benjyw Oct 2, 2017

copumpkin commented Oct 2, 2017

kwlzn Oct 2, 2017

copumpkin Oct 2, 2017

copumpkin commented Oct 2, 2017

benjyw Oct 3, 2017

copumpkin Oct 3, 2017

copumpkin commented Oct 5, 2017

kwlzn left a comment

copumpkin commented Oct 5, 2017

kwlzn commented Oct 5, 2017

copumpkin commented Oct 6, 2017

copumpkin commented Oct 6, 2017

copumpkin commented Oct 9, 2017

copumpkin commented Oct 17, 2017

Allow users to request a fixed python interpreter #4930

Allow users to request a fixed python interpreter #4930

Conversation

copumpkin commented Oct 2, 2017 • edited

Problem

Solution

Result

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin commented Oct 2, 2017

benjyw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin commented Oct 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin commented Oct 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin commented Oct 5, 2017

kwlzn left a comment

Choose a reason for hiding this comment

copumpkin commented Oct 5, 2017

kwlzn commented Oct 5, 2017

copumpkin commented Oct 6, 2017

copumpkin commented Oct 6, 2017

copumpkin commented Oct 9, 2017

copumpkin commented Oct 17, 2017

copumpkin commented Oct 2, 2017 •

edited