pin the PEX_PYTHON{,_PATH} running the pytest pex to avoid using incompatible pytest requirements #7563

cosmicexplorer · 2019-04-15T07:19:11Z

Problem

PytestPrep will generate a pex merging python sources and 3rdparty requirements. This pex will have all of the unique python interpreter constraints from all targets applied to the pex before being built. When executing that pex, it appears that the wrong python interpreter may be selected when the interpreter constraint set is intersecting -- see the added test test_succeeds_for_intersecting_unique_constraints(). This ends up looking like:

Failed to execute PEX file, missing macosx_10_14_x86_64-cp-27-cp27m compatible dependencies for:
more-itertools
pytest
funcsigs
pytest-cov
coverage

Solution

Force the selected interpreter used to create the pytest pex to be used when running it by setting PEX_PYTHON and PEX_PYTHON_PATH.
Add testing for the case of targets with intersecting interpreter constraints (such as ['CPython>=2.7,<4', 'CPython>=3.6']).

Result

Running python tests targets with overlapping interpreter constraints should work!

blorente

This is great, thanks!

wisechengyi

Thanks! would be nice to have some tests if they are not too expensive.

cosmicexplorer · 2019-04-15T16:38:05Z

Thanks! would be nice to have some tests if they are not too expensive.

I think that's reasonable, especially given that some tests are failing in the current CI run. Investigating those failures might be able to produce a new test that checks for the corrected behavior.

jsirois · 2019-04-15T23:17:24Z

src/python/pants/backend/python/tasks/pytest_run.py

+      # Ensure the pex is invoked with the interpreter we selected beforehand to be compatible with
+      # all the relevant targets.
+      interpreter = self.context.products.get_data(PythonInterpreter)
+      env['PEX_PYTHON'] = interpreter.binary


Te pexrc issue seems out of hand and this seems like a bandaid.

Is this more appropriate?

$ pex --help-variables ... PEX_IGNORE_RCFILES: Boolean Explicitly disable the reading/parsing of pexrc files (~/.pexrc). Default: false. ...

It seems like we should be setting this when invoking any pex-based tool in pants, not just here.

I thought that flag was on its way out: pex-tool/pex#673, but if it's okay to use it, then it looks like the thing we want to use.

Yes, the flag, not the option. Flags for pex like --rcfile are only used at build time. options like PEX_IGNORE_RCFILES at runtime. For some options, specifying them at build time or run time both make sense. For this one, only run time makes sense.

Thanks for the clarification!

The flag PEX_IGNORE_RCFILES has not appeared to have an effect on the case I was looking at (and the case I was looking at didn't appear to be due to the use of a pexrc!). I was able to isolate a repro and convert it into the new test_succeeds_for_intersecting_unique_constraints(), which this PR now fixes.

cosmicexplorer · 2019-04-28T03:30:18Z

Refactored this after a while experimenting with the failure we were seeing internally, the result manages to fix the issue without introducing further breakage. It appears the issue was probably not due to a pexrc, and was instead a more general issue, which is now fixed.

This should be ready for review again. Commits are reviewable independently.

cosmicexplorer · 2019-04-28T18:51:05Z

The CI failure is legit, I'm fixing that now.

cosmicexplorer · 2019-04-28T19:06:13Z

Ok, so I just increased the --timeout-default argument for the one failing PytestRun integration test, since the previous failure actually did seem to be flakiness (and it had happened twice in a row). It's suspicious that this change would seem to cause that specific flakiness, and we will be doing performance testing of this change internally to avoid introducing a regression.

jsirois

I think you're still getting lucky here.

See my change to relase.sh here:
https://github.com/pantsbuild/pants/pull/7591/files#diff-9ed7102b7836807dc342cc2246ec4839R92:

pants/build-support/bin/release.sh

Lines 92 to 99 in a6f96ca

    
           # We use `PEX_PYTHON_PATH="${PY}" "${PY}" "${pex}"` instead of either of: 
        
           # 1. PEX_PYTHON_PATH="${PY}" "${pex}" 
        
           # 2. "${PY}" "${pex}" 
        
           # This works around Pex re-exec-ing when it need not and subsequently losing constraints when 
        
           # it need not. The fixes for these are tracked in: 
        
           #   1. https://github.com/pantsbuild/pex/issues/709 
        
           #   2. https://github.com/pantsbuild/pex/issues/710 
        
           PEX_PYTHON_PATH="${PY}" "${PY}" "${pex}" "$@"

That uses PEX_PYTHON_PATH=[python] [python] [pex] ... to execute [pex] ensuring the given [python] interpreter is used. That workaround PR calls out the related pex issues of:

NB: That change does not deal with the possibility of pexrc. However, explicitly setting PEX_PYTHON_PATH overrides any pexrc setting of it. If you have pexrc files that specify both PEX_PYTHON and PEX_PYTHON_PATH, you really should PEX_IGNORE_RCFILES=1 instead of setting both PEX_PYTHON and PEX_PYTHON_PATH explicitly. To me at least - its more clear to say roughly:

run the pex in a hermeticized pex environment (PEX_IGNORE_RCFILES=1)
but mix in our setting for PEX_PYTHON_PATH

NB#2: All the work in this PR is regrettably whac-a-mole. It seems to me it would be a good idea to file an issue to absract out a ~PexExecutor that all of pants can use to hermetically execute a pex using a selected interpreter just like the bits of changed code in this PR eventually will.

…file path

cosmicexplorer · 2019-04-30T02:51:21Z

Ok, the two pex issues you linked seem to match this failure perfectly and the larger context is clear.

NB#2: All the work in this PR is regrettably whac-a-mole. It seems to me it would be a good idea to file an issue to absract out a ~PexExecutor that all of pants can use to hermetically execute a pex using a selected interpreter just like the bits of changed code in this PR eventually will.

Well, in this one I could at least move the functionality into a separate PexExecutor class so other tasks can pick it up when they need it.

NB: That change does not deal with the possibility of pexrc.

I believe the pexrc config is not affecting the failure here, or at least, setting PEX_IGNORE_RCFILES and using the python interpreter in the argv does not work in the way that setting PEX_PYTHON_PATH and PEX_PYTHON does. On my laptop the changes in 5a461db (setting just PEX_PYTHON_PATH and explicitly selecting the interpter) caused the pytest invocation to fail before tests were run:

============== test session starts ===============
platform darwin -- Python 3.6.4, pytest-3.6.4, py-1.8.0, pluggy-0.7.1 -- /Users/dmcclanahan/tools/pants/build-support/pants_dev_deps.py36.venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/dmcclanahan/tools/pants, inifile: /Users/dmcclanahan/tools/pants/.pants.d/test/pytest-prep/CPython-3.6.4/667aa724ef06c0a4cda5a996a39591b15fbcde5a/pytest.ini
plugins: cov-2.4.0, timeout-1.2.1
collecting ...
- generated xml file: /Users/dmcclanahan/tools/pants/.pants.d/test/pytest/tests.python.pants_test.backend.python.tasks.pytest_run/junitxml/TEST-tests.python.pants_test.backend.python.tasks.pytest_run.xml -
========== no tests ran in 0.04 seconds ==========
ERROR: not found: /Users/dmcclanahan/tools/pants/build-support/pants_dev_deps.py36.venv/bin/python3
(no name '/Users/dmcclanahan/tools/pants/build-support/pants_dev_deps.py36.venv/bin/python3' in any of [])


                   tests/python/pants_test/backend/python/tasks:pytest_run    .....   NOT RUN

If you have pexrc files that specify both PEX_PYTHON and PEX_PYTHON_PATH

I do. Unfortunately setting PEXRC_IGNORE_FILES=1 in 5a461db failed to run tests, while explicitly setting both PEX_PYTHON_PATH and PEX_PYTHON successfully ran tests.

I think regardless of the eventual fix, generalizing it into a PexExecutor to pin the interpreter seems like a good idea.

jsirois · 2019-04-30T04:31:26Z

Well, in this one I could at least move the functionality into a separate PexExecutor class so other tasks can pick it up when they need it.

I encourage you to fight your better nature here and maintain scope. It's not just for you, it's for reviewers. It's nice for PRs to close out ~promptly as the norm. I'd just file an issue.

…rpreter file path" This reverts commit 9a68c3c.

cosmicexplorer · 2019-04-30T17:24:09Z

@jsirois I've broken out abstracting hermetic pex execution into #7640, and while I haven't determined why using PEX_PYTHON_PATH=... /path/to/python /path/to/pex.pex args... (attempted in 9a68c3c) is failing, the rationale for instead having to set both PEX_PYTHON and PEX_PYTHON_PATH is described in the docstring for _constrain_pytest_interpreter_search_path(). The added test demonstrates a case which is currently broken that is now fixed, and this diff fixes the same breakage we were seeing internally. Is it reasonable to try to merge this PR now, or am I missing anything?

stuhood

Would clarify and document the argument to create_pex.

stuhood · 2019-04-30T18:42:26Z

src/python/pants/backend/python/tasks/python_execution_task_base.py

+        }
+        self.context.log.debug('unique_constraints:\n{}'.format(unique_constraints))
+
+        if pin_selected_interpreter:


This flag seems to disable the unique_constraints codepath entirely. should those be computed under that branch of the if instead?

It seems like rather than "pinning", this is more like: "use_global_python_interpreter" or something, because what it pins to is specifically the PythonInterpreter product. It should definitely have a mention in the docstring.

I've removed the extra argument to create_pex() in favor of removing the unique_constraints code path entirely, as every consumer of this method is always going to depend on requirements pexes resolved specifically for the single global selected interpreter. I've also added a docstring clarifying all of this.

As noted in #7563 (comment), I've added back the pin_selected_interpreter argument, but added a descriptive docstring. I'll be following up with a change that removes this.

…ex()

src/python/pants/backend/python/tasks/pytest_run.py

tests/python/pants_test/task_test_base.py

cosmicexplorer · 2019-05-01T04:00:42Z

I'm going to revert the changes to create_pex() which are causing the one CI failure and break it out into a followup PR.

…create_pex()" This reverts commit 15195d3.

src/python/pants/backend/python/tasks/python_execution_task_base.py

### Problem The problems observed in #7775 and #7563 had a third (and hopefully final: I'll audit before we land this) buddy in the `PythonRun` case. The tests in https://github.com/pantsbuild/pants/blob/817f7f6aedad8143fe7b2cf86559019254230da4/tests/python/pants_test/backend/python/tasks/test_python_run_integration.py#L134-L184 attempt to cover this case, but they were not using mixed contexts (contexts containing `(2-or-3, 3-only)` and `(2-only, 2-or-3)` constraints). ### Solution Similar to #7775 and #7563, do not include transitive constraints when `./pants run`ing a binary. Clarify the documentation of `PythonExecutionTaskBase`. Expand coverage of the mixed context case in existing tests. ### Result The tests linked above fail before this change for `PythonRun`, and succeed afterward.

The problems observed in #7775 and #7563 had a third (and hopefully final: I'll audit before we land this) buddy in the `PythonRun` case. The tests in https://github.com/pantsbuild/pants/blob/817f7f6aedad8143fe7b2cf86559019254230da4/tests/python/pants_test/backend/python/tasks/test_python_run_integration.py#L134-L184 attempt to cover this case, but they were not using mixed contexts (contexts containing `(2-or-3, 3-only)` and `(2-only, 2-or-3)` constraints). Similar to #7775 and #7563, do not include transitive constraints when `./pants run`ing a binary. Clarify the documentation of `PythonExecutionTaskBase`. Expand coverage of the mixed context case in existing tests. The tests linked above fail before this change for `PythonRun`, and succeed afterward.

cosmicexplorer requested review from jsirois, wisechengyi and blorente April 15, 2019 07:19

blorente approved these changes Apr 15, 2019

View reviewed changes

wisechengyi approved these changes Apr 15, 2019

View reviewed changes

jsirois reviewed Apr 15, 2019

View reviewed changes

cosmicexplorer force-pushed the pytest-as-python-tool branch 2 times, most recently from cadf7d2 to 471bbf3 Compare April 28, 2019 03:13

cosmicexplorer changed the title ~~fix the PEX_PYTHON of the pytest pex when run to avoid pexrc clashing~~ fix the PEX_PYTHON of the pytest pex when run to avoid using incompatible pytest requirements Apr 28, 2019

cosmicexplorer force-pushed the pytest-as-python-tool branch from 471bbf3 to 75564dc Compare April 28, 2019 03:25

cosmicexplorer force-pushed the pytest-as-python-tool branch 2 times, most recently from e3ac73e to c2f39ad Compare April 28, 2019 16:48

cosmicexplorer requested review from wisechengyi, blorente, stuhood, illicitonion and ity April 28, 2019 18:09

cosmicexplorer changed the title ~~fix the PEX_PYTHON of the pytest pex when run to avoid using incompatible pytest requirements~~ pin the PEX_PYTHON{,_PATH} running the pytest pex to avoid using incompatible pytest requirements Apr 28, 2019

jsirois reviewed Apr 29, 2019

View reviewed changes

stuhood added this to the 1.16.x milestone Apr 29, 2019

cosmicexplorer added 3 commits April 29, 2019 19:19

refactor the pytest run task test base declaratively

88602c0

pin the pytest pex interpreter path to the selected interpreter

b99bcd4

increase the timeout default for the success test to fix flakiness

bd6fdb1

cosmicexplorer force-pushed the pytest-as-python-tool branch from 9d294f8 to 5a461db Compare April 30, 2019 02:43

attempt to refactor the pytest pex invocation to use the interpreter …

9a68c3c

…file path

cosmicexplorer force-pushed the pytest-as-python-tool branch from 5a461db to 9a68c3c Compare April 30, 2019 02:48

cosmicexplorer mentioned this pull request Apr 30, 2019

abstract out a PexExecutor class to hermetically execute a pex (with a pinned interpreter) #7640

Closed

Revert "attempt to refactor the pytest pex invocation to use the inte…

1dd1d8f

…rpreter file path" This reverts commit 9a68c3c.

stuhood approved these changes Apr 30, 2019

View reviewed changes

always only apply a single == interpreter constraint in self.create_p…

15195d3

…ex()

jsirois approved these changes Apr 30, 2019

View reviewed changes

src/python/pants/backend/python/tasks/pytest_run.py Outdated Show resolved Hide resolved

tests/python/pants_test/task_test_base.py Show resolved Hide resolved

add PEX_IGNORE_RCFILES=1

dfef85c

cosmicexplorer force-pushed the pytest-as-python-tool branch from 1f56f37 to dfef85c Compare May 1, 2019 02:53

cosmicexplorer mentioned this pull request May 1, 2019

allow DeclarativeTaskTestMixin.TaskInvocationResult to represent a failure result #7644

Closed

fix other review comments

79bee69

Revert "always only apply a single == interpreter constraint in self.…

538a930

…create_pex()" This reverts commit 15195d3.

jsirois approved these changes May 1, 2019

View reviewed changes

src/python/pants/backend/python/tasks/python_execution_task_base.py Outdated Show resolved Hide resolved

add docstring for create_pex()

ac20482

cosmicexplorer force-pushed the pytest-as-python-tool branch from 36d15cd to ac20482 Compare May 1, 2019 16:41

cosmicexplorer merged commit 823ae84 into pantsbuild:master May 1, 2019

cosmicexplorer mentioned this pull request May 9, 2019

[WIP] add prepend_to_pythonpath=True workaround for some 3rdparty python packages #7695

Closed

stuhood mentioned this pull request May 24, 2019

Always use the global interpreter for PythonExecutionTaskBase. #7801

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pin the PEX_PYTHON{,_PATH} running the pytest pex to avoid using incompatible pytest requirements #7563

pin the PEX_PYTHON{,_PATH} running the pytest pex to avoid using incompatible pytest requirements #7563

cosmicexplorer commented Apr 15, 2019 •

edited

blorente left a comment

wisechengyi left a comment

cosmicexplorer commented Apr 15, 2019

jsirois Apr 15, 2019

blorente Apr 16, 2019

jsirois Apr 22, 2019

blorente Apr 23, 2019

cosmicexplorer Apr 28, 2019

cosmicexplorer commented Apr 28, 2019

cosmicexplorer commented Apr 28, 2019

cosmicexplorer commented Apr 28, 2019

jsirois left a comment •

edited

cosmicexplorer commented Apr 30, 2019

jsirois commented Apr 30, 2019 •

edited

cosmicexplorer commented Apr 30, 2019 •

edited

stuhood left a comment

stuhood Apr 30, 2019

cosmicexplorer Apr 30, 2019

cosmicexplorer May 1, 2019

cosmicexplorer commented May 1, 2019

	# We use `PEX_PYTHON_PATH="${PY}" "${PY}" "${pex}"` instead of either of:
	# 1. PEX_PYTHON_PATH="${PY}" "${pex}"
	# 2. "${PY}" "${pex}"
	# This works around Pex re-exec-ing when it need not and subsequently losing constraints when
	# it need not. The fixes for these are tracked in:
	# 1. https://github.com/pantsbuild/pex/issues/709
	# 2. https://github.com/pantsbuild/pex/issues/710
	PEX_PYTHON_PATH="${PY}" "${PY}" "${pex}" "$@"

pin the PEX_PYTHON{,_PATH} running the pytest pex to avoid using incompatible pytest requirements #7563

pin the PEX_PYTHON{,_PATH} running the pytest pex to avoid using incompatible pytest requirements #7563

Conversation

cosmicexplorer commented Apr 15, 2019 • edited

Problem

Solution

Result

blorente left a comment

Choose a reason for hiding this comment

wisechengyi left a comment

Choose a reason for hiding this comment

cosmicexplorer commented Apr 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Apr 28, 2019

cosmicexplorer commented Apr 28, 2019

cosmicexplorer commented Apr 28, 2019

jsirois left a comment • edited

Choose a reason for hiding this comment

cosmicexplorer commented Apr 30, 2019

jsirois commented Apr 30, 2019 • edited

cosmicexplorer commented Apr 30, 2019 • edited

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented May 1, 2019

cosmicexplorer commented Apr 15, 2019 •

edited

jsirois left a comment •

edited

jsirois commented Apr 30, 2019 •

edited

cosmicexplorer commented Apr 30, 2019 •

edited