New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect ns packages using correct interpreter. #6428

Merged
merged 2 commits into from Sep 6, 2018

Conversation

Projects
None yet
2 participants
@jsirois
Copy link
Member

jsirois commented Aug 31, 2018

Previously __init__.pys were parsed in the context of the ambient
interpreter executing the SetupPy task which could lead to parse
failures for new or removed language features used in those
__init__.pys. We now construct a PEX using the appropriate interpreter
and use it to run ns package detection.

Fixes #6415

@stuhood
Copy link
Member

stuhood left a comment

Thanks John!

chroot = os.path.join(self.workdir, 'nsutil', interpreter.version_string)
if not os.path.exists(chroot):
pex_info = PexInfo.default(interpreter=interpreter)
with safe_concurrent_creation(chroot) as scratch:

This comment has been minimized.

@stuhood

stuhood Aug 31, 2018

Member

Is "anonymous pex for code string" utility function worthy?

This comment has been minimized.

@jsirois

jsirois Aug 31, 2018

Member

Perhaps. I think #6401 will force the issue and require similar treatment of our generation of setup.py.

@jsirois jsirois force-pushed the jsirois:issues/6415 branch from ed731dc to 59188a3 Sep 2, 2018

@jsirois

This comment has been minimized.

Copy link
Member

jsirois commented Sep 2, 2018

OK - python3.6 issues are due to an executable /opt/pyenv/shims/python3.6 being found on the trusty image, but that executable not actually working: travis-ci/travis-ci#8315

Thinking how best to handle this...

@jsirois jsirois force-pushed the jsirois:issues/6415 branch 3 times, most recently from b5c8819 to b924fec Sep 2, 2018

jsirois added a commit to jsirois/pants that referenced this pull request Sep 2, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Specify `python: "2.7"` which floats us to the most modern python 2.7
  (2.7.14 currently)
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this pull request Sep 3, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to `python: "2.7.14".
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this pull request Sep 4, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this pull request Sep 4, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this pull request Sep 5, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this pull request Sep 6, 2018

Tighten travis matrix and python activation.
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit that referenced this pull request Sep 6, 2018

Tighten travis matrix and python activation. (#6440)
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support #6428 which fixes #6415.

jsirois added some commits Aug 31, 2018

Detect ns packages using correct interpreter.
Previously `__init__.py`s were parsed in the context of the ambient
interpreter executing the `SetupPy` task which could lead to parse
failures for new or removed language features used in those
`__init__.py`s. We now construct a PEX using the appropriate interpreter
and use it to run ns package detection.

Fixes #6415

@jsirois jsirois force-pushed the jsirois:issues/6415 branch from b924fec to 40f7730 Sep 6, 2018

@jsirois jsirois merged commit f83970f into pantsbuild:master Sep 6, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jsirois jsirois deleted the jsirois:issues/6415 branch Sep 6, 2018

jsirois added a commit to jsirois/pants that referenced this pull request Sep 7, 2018

Tighten travis matrix and python activation. (pantsbuild#6440)
A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support pantsbuild#6428 which fixes pantsbuild#6415.

jsirois added a commit to jsirois/pants that referenced this pull request Sep 7, 2018

Detect ns packages using correct interpreter. (pantsbuild#6428)
Previously `__init__.py`s were parsed in the context of the ambient
interpreter executing the `SetupPy` task which could lead to parse
failures for new or removed language features used in those
`__init__.py`s. We now construct a PEX using the appropriate interpreter
and use it to run ns package detection.

Fixes pantsbuild#6415

@jsirois jsirois referenced this pull request Sep 7, 2018

Merged

Cherry-pick setup_py fixes #6472

jsirois added a commit that referenced this pull request Sep 10, 2018

Cherry-pick setup_py fixes (#6472)
* [fix] Pass full path to isdir rather than just basename. (#6453)

* [fix] Pass full path to isdir rather than just basename.

* [test] Add tests for fix.

* Tighten travis matrix and python activation. (#6440)

A few improvements and fixes to the travis setup:
+ Upgrade to to python 2.7.14 via `python: "2.7"`.
+ Use `name` uniformly to name shards and kill the `SHARD` env var which
  was only used to achieve the same end.
+ Use anchors to DRY up python version and clippy shard failure
  ignoring.
+ Ensure travis `pyenv` pythons are on `PATH`. This works around
  travis-ci/travis-ci#8315 and nets us both
  python 2.7 and python 3.6 on all `language: python` shards.

This last is needed to support #6428 which fixes #6415.

* Remove broken pyenv shims from the PATH. (#6469)

It turns out these would still get found in various contexts since the
interpreter cache tries to setup everything it finds on the PATH. Add
improved context in the wrapper script to show exactly which pythons
Pants will have access to.

* Fixup tests involving pexrc. (#6446)

Previously these tests weren't reliably reading a pexrc and had fragile
workarounds and paper-overs as a result. pexrc_util now reliably injects
a pexrc with a temporary HOME.

Also cleanup the affected tests - use @skipIf, respect 100 cols, ensure
path comparisons with judicious application of `os.realpath`.

* Detect ns packages using correct interpreter. (#6428)

Previously `__init__.py`s were parsed in the context of the ambient
interpreter executing the `SetupPy` task which could lead to parse
failures for new or removed language features used in those
`__init__.py`s. We now construct a PEX using the appropriate interpreter
and use it to run ns package detection.

Fixes #6415

* Fix setup.py rendering. (#6439)

We now render setup.py from both python 2 & 3 such that the result
is ingestible by both python 2 & 3.

Fixes #6401
Fixes #6392
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment