Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pex 1.5.1 bombs with unhandled pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName on certain resolves #615

Closed
kwlzn opened this issue Nov 2, 2018 · 5 comments · Fixed by #617
Closed

pex 1.5.1 bombs with unhandled pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName on certain resolves #615

kwlzn opened this issue Nov 2, 2018 · 5 comments · Fixed by #617
Assignees
@jsirois
Labels
bug

Comments

@kwlzn
Copy link
Contributor

kwlzn commented Nov 2, 2018 •
edited
Loading

repro:

[omerta show]$ wget -q https://github.com/pantsbuild/pex/releases/download/v1.5.1/pex27 -O pex27-1.5.1
[omerta show]$ ./pex27-1.5.1 apache-beam[gcp]==2.7.0
Traceback (most recent call last):
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/pex.py", line 349, in execute
    working_set = self._activate()
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/pex.py", line 90, in _activate
    for dist in env.activate():
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/environment.py", line 152, in activate
    self._working_set = self._activate()
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/environment.py", line 211, in _activate
    resolved = self._resolve(working_set, all_reqs)
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/environment.py", line 165, in _resolve
    if req.marker and not req.marker.evaluate(environment=self._target_interpreter_env):
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/pkg_resources/_vendor/packaging/markers.py", line 301, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/pkg_resources/_vendor/packaging/markers.py", line 226, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/pkg_resources/_vendor/packaging/markers.py", line 208, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

vs 1.4.4:

[omerta show]$ wget -q https://github.com/pantsbuild/pex/releases/download/v1.4.4/pex27 -O pex27-1.4.4
[omerta show]$ ./pex27-1.4.4 apache-beam[gcp]==2.7.0
Python 2.7.15 (default, Jun 20 2018, 17:40:09) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
@kwlzn kwlzn mentioned this issue Nov 2, 2018
Closed
@kwlzn
Copy link
Contributor Author

kwlzn commented Nov 2, 2018

noting the shape of the markers in the failure case is (from the apache-beam wheel):

Provides-Extra: gcp
Requires-Dist: google-apitools (<=0.5.20,>=0.5.18); extra == 'gcp'
Requires-Dist: proto-google-cloud-datastore-v1 (<=0.90.4,>=0.90.0); extra == 'gcp'
Requires-Dist: googledatastore (==7.0.1); extra == 'gcp'
Requires-Dist: google-cloud-pubsub (==0.26.0); extra == 'gcp'
Requires-Dist: proto-google-cloud-pubsub-v1 (==0.15.4); extra == 'gcp'
Requires-Dist: google-cloud-bigquery (==0.25.0); extra == 'gcp'

@jsirois
Copy link
Member

jsirois commented Nov 2, 2018

OK - thanks for the report. Easy repro and the fix appears to be tiny.

@jsirois jsirois removed the resolver label Nov 2, 2018
@jsirois
Copy link
Member

jsirois commented Nov 2, 2018

The bug is not on the resolution side, but in the activation side where we filter out embedded dists not appropriate to the current enviornment the pex is running in.

@jsirois jsirois self-assigned this Nov 2, 2018
jsirois added a commit to jsirois/pex that referenced this issue Nov 3, 2018
@jsirois
Fixup PEXEnvironment extras resolution.
f47e3fa 
In pex-tool#582 and pex-tool#592 support for environment markers was added to the pex
runtime. In so doing, the resolver was fixed to record full requirement
strings into PEX-INFO. Since part of those full requirement strings
could now include environment markers that selected for active extras,
a bug was introduced since we did not also add the active extras to the
environment marker evaluation environment.

This change adds a failing test that is fixed by properly setting up the
environment marker environment to include active extras.

Fixes pex-tool#615.
@jsirois jsirois mentioned this issue Nov 3, 2018
Merged
jsirois added a commit to jsirois/pex that referenced this issue Nov 3, 2018
@jsirois
Fixup PEXEnvironment extras resolution.
5c3de4c 
In pex-tool#582 and pex-tool#592 support for environment markers was added to the pex
runtime. In so doing, the resolver was fixed to record full requirement
strings into PEX-INFO. Since part of those full requirement strings
could now include environment markers that selected for active extras,
a bug was introduced since we did not also add the active extras to the
environment marker evaluation environment.

This change adds a failing test that is fixed by properly setting up the
environment marker environment to include active extras.

Fixes pex-tool#615.
@jsirois jsirois mentioned this issue Nov 3, 2018
Closed
jsirois added a commit that referenced this issue Nov 4, 2018
@jsirois
Fixup PEXEnvironment extras resolution. (#617)
e53fadc 
In #582 and #592 support for environment markers was added to the pex
runtime. In so doing, the resolver was fixed to record full requirement
strings into PEX-INFO. Since part of those full requirement strings
could now include environment markers that selected for active extras,
a bug was introduced since we did not also add the active extras to the
environment marker evaluation environment.

This change adds a failing test that is fixed by properly setting up the
environment marker environment to include active extras.

Fixes #615.
@jsirois
Copy link
Member

jsirois commented Nov 16, 2018

I'm going to re-open this one since the current workaround fix just pushes a new bug downstream and the fix in #613 will handle both these issues.

@jsirois jsirois reopened this Nov 16, 2018
@jsirois
Copy link
Member

jsirois commented Nov 20, 2018

Opps - wrong re-open.

@jsirois jsirois closed this as completed Nov 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixup PEXEnvironment extras resolution. jsirois/pex
2 participants
@jsirois @kwlzn