New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

20.5.0+ breaks existing setup.py files - UndefinedEnvironmentName #523

Closed
lnielsen opened this Issue Mar 29, 2016 · 50 comments

Comments

Projects
None yet
@lnielsen

lnielsen commented Mar 29, 2016

The latest 20.6.4 PyPI is breaking our setup.py files with the following exception:

Traceback (most recent call last):
  File "setup.py", line 163, in <module>
    cmdclass={'test': PyTest},
  File "/opt/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/command/test.py", line 152, in run
    self.distribution.fetch_build_eggs(self.distribution.tests_require)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/__init__.py", line 814, in resolve
    if req.marker and not req.marker.evaluate():
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 273, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 198, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 180, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

@lnielsen lnielsen changed the title from 20.6.4 breaks existing setup.py files - UndefinedEnvironmentName to 20.5.0+ breaks existing setup.py files - UndefinedEnvironmentName Mar 29, 2016

@lnielsen

This comment has been minimized.

lnielsen commented Mar 29, 2016

Seems to be related to this commit d25e2a5

@jaraco

This comment has been minimized.

Member

jaraco commented Mar 29, 2016

@s-t-e-v-e-n-k Can you look into this one?

@lnielsen How severe is the failure? Should I pull 20.5+ from PyPI?

@lnielsen

This comment has been minimized.

lnielsen commented Mar 29, 2016

Looks pretty serve. Just upgrading setuptools in a virtualenv breaks the code.

@lnielsen

This comment has been minimized.

lnielsen commented Mar 29, 2016

Just debugging now to provide some more information.

@jaraco

This comment has been minimized.

Member

jaraco commented Mar 29, 2016

I've removed 20.6.4 and 20.5.0 until the issue can be resolved.

@msabramo

This comment has been minimized.

Contributor

msabramo commented Mar 29, 2016

Also seeing this.

@msabramo

This comment has been minimized.

Contributor

msabramo commented Mar 29, 2016

My colleague has been looking into this and he thinks that it has to do with the fact that our tests are using a variable/parameter name called extra_environ. This is a parameter that is used by webtest (e.g.: see http://docs.pylonsproject.org/projects/webtest/en/latest/testapp.html#modifying-the-environment-simulating-authentication)

@bszonye

This comment has been minimized.

bszonye commented Mar 29, 2016

I am seeing this in code that has variables and parameters named extra_environ that are ultimately passed to the webtest API. Renaming to extra_env_vars and extra_evars still hit the problem, but renaming to other_environ fixed it, so it looks like something is incorrectly identifying the extra_ pattern as an environment marker.

@nakato

This comment has been minimized.

Contributor

nakato commented Mar 30, 2016

@lnielsen I can't reproduce this. How are you getting that traceback? What commands are you executing? What is the logging leading up to it? Is there anything that follows it?

@jaraco

This comment has been minimized.

Member

jaraco commented Mar 31, 2016

I can't replicate this either.

I inferred from the link that the command being executed is python setup.py test in the invenio-records-rest project, so I checked out that repo and ran setup.py test under Python 2.7 and 3.5 using setuptools 20.5+, and while some tests failed, the reported error did not occur.

I sense now that the failures are fairly isolated and only affecting tests, so I'm going to re-release the code in the latest version, but I am eager to see a fix for this issue, so do please provide more detail to describe how one can recreate the issue, preferably with a minimal use case.

@bszonye

This comment has been minimized.

bszonye commented Mar 31, 2016

Our teams are encountering this when using the webtest API in a py.test fixture, so it affects all of their tests. I do not yet have a workaround that successfully builds against the webtest classes without triggering the setuptools bug, so those teams cannot even run commit tests to integrate new code.

@jaraco

This comment has been minimized.

Member

jaraco commented Mar 31, 2016

@bszonye Sorry. I don't mean to cause you trouble. Can you pin your environment to Setuptools 20.4 until the issue can be resolved?

@bszonye

This comment has been minimized.

bszonye commented Mar 31, 2016

That’s what I recommended to the team for now, to pin setuptools<20.5 – tomorrow I will take a closer look to see if I can find a minimal test case.

@bszonye

This comment has been minimized.

bszonye commented Mar 31, 2016

@jaraco Just confirmed with the team that version 20.6.6 does break their tests again, but they are able to work around by pinning to the 20.4 version. I suspect that something in packaging is misidentifying extra markers from code that happens to contain the string in certain identifiers.

Our traceback ends at the same place but starts with py.test instead of setup.py test:

[tests.python.functional.test_XXX.TestXXX.test_XXX] tests/python/functional/utils.py:44: in setUp
    self.testapp = get_webtest_testapp()
tests/python/functional/utils.py:98: in get_webtest_testapp
    app = get_app(join(conf_dir, 'test.ini'))
.tox/py27/local/lib/python2.7/site-packages/pyramid/paster.py:31: in get_app
    global_conf=options)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:247: in loadapp
    return loadobj(APP, uri, name=name, **kw)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:271: in loadobj
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:296: in loadcontext
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:320: in _loadconfig
    return loader.get_context(object_type, name, global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:450: in get_context
    global_additions=global_additions)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:559: in _pipeline_app_context
    APP, pipeline[-1], global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:454: in get_context
    section)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:476: in _context_from_use
    object_type, name=use, global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:406: in get_context
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:296: in loadcontext
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:328: in _loadegg
    return loader.get_context(object_type, name, global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:620: in get_context
    object_type, name=name)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:640: in find_egg_entry_point
    pkg_resources.require(self.spec)
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py:947: in require
    needed = self.resolve(parse_requirements(requirements))
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py:814: in resolve
    if req.marker and not req.marker.evaluate():
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py:278: in evaluate
    return _evaluate_markers(self._markers, current_environment)
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py:203: in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py:185: in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
E   UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
@s-t-e-v-e-n-k

This comment has been minimized.

Contributor

s-t-e-v-e-n-k commented Mar 31, 2016

@bszonye I'd like to see which requirement is causing this issue, are you able to edit the virtual env's copy of local/lib/python2.7/site-packages/pkg_resources/init.py to log what 'req' is set to just before the call to req.marker.evaluate()? I'd personally use the q module.

@warner

This comment has been minimized.

warner commented Mar 31, 2016

This is biting us too, on today's (brand-new) release of tahoe-lafs. To reproduce: pip install tahoe-lafs, then tahoe --version.

If I add a print req just before the req.marker.evaluate() call at pkg_resources/__init__.py line 814, then when I run our application, I see:

pyOpenSSL>=0.14
Nevow>=0.11.1
Twisted[tls]>=15.1.0
pyasn1-modules>=0.0.5
pyasn1>=0.1.8
characteristic>=14.0.0
service-identity
pycryptopp>=0.6.0
pycrypto!=2.2,!=2.4,>=2.1.0
foolscap>=0.10.1
zope.interface!=3.6.3,!=3.6.4,>=3.6.0
simplejson>=1.4
zfec>=1.1.0
setuptools>=11.3
six>=1.5.2
cryptography>=1.3
twisted>=13.0
pyopenssl>=0.13; extra == "tls"

followed by the same traceback as above.

Tahoe depends, among most of those other things, on twisted[tls] >= 15.1.0 and pyopenssl >= 0.14. (See https://tahoe-lafs.org/~warner/deps.png for a graph of the dependencies, created by https://github.com/tahoe-lafs/tahoe-lafs/blob/master/misc/coding_tools/graph-deps.py).

Nevow wants twisted >= 0.13, so I'm guessing that's where the next-to-last req comes from. Twisted[tls] wants pyopenssl >= 0.13. Is it normal that the req should have "extra == tls" on the pyopenssl line, when the extra is associated with Twisted?

Grepping through our code, we do have functions and variables with extra_ in them, in case it matters.

Please let me know if there's anything I can do to help track this one down.

@warner

This comment has been minimized.

warner commented Mar 31, 2016

I don't know this code very well, but I think the requirement that it's tripping on is this one from Twisted's dist-info/METADATA file:

Requires-Dist: pyopenssl (>=0.13); extra == 'tls'

and I think the req.marker.evaluate() call in the traceback is treating this extra=='tls' as a marker (e.g. python_version>=2.7), rather than as an extra.

I know the line-based METADATA format necessarily merges extras and markers into the same syntax. Is this causing confusion? I see calls to e.g. req.marker.evaluate({'extra': extra}) elsewhere, and it seems like adding that kind of argument to our evaluate() call would help. I see req.extras is available, but I'm not sure how to deal with the fact that it's a list. Call it once with everything in req.extras and continue if any of them say True? Or if none of them say False?

@s-t-e-v-e-n-k

This comment has been minimized.

Contributor

s-t-e-v-e-n-k commented Mar 31, 2016

extra == 'tls' is a valid environment marker. But yes, you're totally right, the evaluate() call needs to respect extras, which I'll be looking at tomorrow.

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented Mar 31, 2016

For clarity, extra=='tls' is not a valid marker for setup.py scripts. Its valid in the syntax for distribution metadata only. Callers of the Python API must supply a extra mapping when evaluating requirements specifications if they are processing distribution metadata entries (like the one above) .

tl;dr: pkg_resources, when processing METADATA files can supply a value for extra. It may be wired up wrongly at the moment. extra can't be used from setup_requires or other such places though - its a gnarly thing in the distirbution metadata spec, not a user interface element.

I'll look more closely at the actual calling code tomorrow - this is mainly meant to hint to folk that have already been looking at it.

@rbtcollins

This comment has been minimized.

Contributor

rbtcollins commented Mar 31, 2016

Oh, and - 23:19 < warner> we've got a requirement on "twisted[tls]", and twisted"s got an extra that says [tls] needs pyopenssl
23:19 < warner> and the exception happens when it tries to do req.marker.evaluate() on a req for pyopenssl and a marker for "extra = 'tls'"
23:21 < lifeless> yeah
23:22 < lifeless> I think the problem here is that something has started reflecting markers back into requiremnets
23:22 < lifeless> but we can only do that for a subset of markers
23:22 < lifeless> the extra mapping needs to be translated when its mapped back

from IRC.

HTH

@autopulated

This comment has been minimized.

autopulated commented Mar 31, 2016

Also just tripped over this, to reproduce:

pip install requests[security]
python -c "import pkg_resources; pkg_resources.require('requests[security]')"

Reverting back to an older setuptools with pip install setuptools==20.4 works.

@jaraco

This comment has been minimized.

Member

jaraco commented Mar 31, 2016

Thanks @autopulated - I was able to reproduce the issue with those commands. This definitely is a widespread issue. Rather than pull the latest releases, I'm going to disable this new functionality until a solution can be implemented.

@robdennis

This comment has been minimized.

robdennis commented Mar 31, 2016

I'm seeing this also when running a console entry point:
leaving this stack trace in case it helps anyone. I got it on both versions 20.5 and 20.6.6 (we don't pin setuptools versions)

I happened when executing an entry point, here's the setup.py definition:

      entry_points={'console_scripts':
                    ['redacted_name = redacted_name.main:main'],
                    },

and the stack trace:

File "/opt/redacted_name/bin/redacted_name", line 8, in <module>
    from pkg_resources import load_entry_point
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2905, in <module>
    @_call_aside
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2891, in _call_aside
    f(*args, **kwargs)
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2918, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 642, in _build_master
    ws.require(__requires__)
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 947, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 814, in resolve
    if req.marker and not req.marker.evaluate():
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 273, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/opt/yamato/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 198, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 180, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

jaraco added a commit that referenced this issue Mar 31, 2016

@jaraco

This comment has been minimized.

Member

jaraco commented Mar 31, 2016

v20.6.7 is released bypassing the issue.

@bszonye

This comment has been minimized.

bszonye commented Mar 31, 2016

Thanks for the troubleshooting tips, @s-t-e-v-e-n-k – I’ll see which requirement is causing the problem for us. I now suspect that it’s something dynamically loaded in Pyramid, and the extra_environ identifiers are a red herring.

@gitrookie

This comment has been minimized.

gitrookie commented May 17, 2016

I am getting this same issue.

pip install scikit-image
Collecting scikit-image
  Using cached scikit_image-0.12.3-cp35-cp35m-manylinux1_x86_64.whl
Requirement already satisfied (use --upgrade to upgrade): pillow>=2.1.0 in /usr/lib/python3/dist-packages (from scikit-image)
Requirement already satisfied (use --upgrade to upgrade): six>=1.7.3 in /usr/lib/python3/dist-packages (from scikit-image)
Collecting dask[array]>=0.5.0 (from scikit-image)
  Using cached dask-0.9.0-py2.py3-none-any.whl
Collecting networkx>=1.8 (from scikit-image)
  Using cached networkx-1.11-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): numpy; extra == "array" in /usr/lib/python3/dist-packages (from dask[array]>=0.5.0->scikit-image)
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 328, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 748, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 360, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 448, in _prepare_file
    req_to_install, finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 387, in _check_skip_installed
    req_to_install.check_if_exists()
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 997, in check_if_exists
    self.satisfied_by = pkg_resources.get_distribution(self.req)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 535, in get_distribution
    dist = get_provider(dist)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 415, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources.extern.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

How to circumvent it. Thanks
@jaraco

This comment has been minimized.

Member

jaraco commented Aug 2, 2016

@gitrookie: What version of setuptools do you have? Looks like you have a wheel called pkg_resources-0.0.0, which I think is an unreleased and unsupported version of pkg_resources.

@vhakulin

This comment has been minimized.

vhakulin commented Sep 28, 2016

This seems to be broken again in setuptools 28.0.0: I was generating message files with pybabel from jinja templates when I got this exception.

Downgrading setuptools to 27.3.1 fixed the issue.

@zgoda

This comment has been minimized.

zgoda commented Nov 9, 2016

Still experiencing this with 28.8.0 when calling PyBabel (27.3.1 works fine):

$ pybabel extract -F ./babel.cfg -k lazy_gettext -o ./ecpapp/translations/messages.pot ./
[ ... ]
extracting messages from ecpapp/templates/base.html (extensions="jinja2.ext.autoescape,jinja2.ext.with_", encoding="utf-8")
Traceback (most recent call last):
  File "/home/jazg/v/emcp/bin/pybabel", line 11, in <module>
    sys.exit(main())
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 911, in main
    return CommandLineInterface().run(sys.argv)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 836, in run
    return cmdinst.run()
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 454, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 156, in extract_from_dir
    dirpath=absname,
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    strip_comment_tags=strip_comment_tags
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 236, in extract_from_file
    strip_comment_tags))
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 298, in extract
    func = entry_point.load(require=True)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
@jaraco

This comment has been minimized.

Member

jaraco commented Dec 6, 2016

Downgrading setuptools to 27.3.1 fixed the issue.

So here's the diff from 27.3.1 to 28.0.0. I don't see anything there that appears relevant to the issue at hand. The one change in pkg_resources is only dealing with encoding issues.

Using the commands where I was able to replicate the issue earlier, I'm unable to replicate it now with Setuptools 28.0.0:

$ python -m venv ~/.envs/issue523
$ ~/.envs/issue523/bin/python -m pip install setuptools==28.0.0
Collecting setuptools==28.0.0
  Downloading setuptools-28.0.0-py2.py3-none-any.whl (467kB)
    100% |████████████████████████████████| 471kB 2.6MB/s 
Installing collected packages: setuptools
Successfully installed setuptools-28.8.0
$ ~/.envs/issue523/bin/python -m pip install 'requests[security]'
Collecting requests[security]
  Using cached requests-2.12.3-py2.py3-none-any.whl
Collecting cryptography>=1.3.4; extra == "security" (from requests[security])
  Using cached cryptography-1.6.tar.gz
Requirement already satisfied: idna>=2.0.0; extra == "security" in ./.envs/issue523/lib/python3.6/site-packages (from requests[security])
Collecting pyOpenSSL>=0.14; extra == "security" (from requests[security])
  Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.8 in ./.envs/issue523/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Requirement already satisfied: six>=1.4.1 in ./.envs/issue523/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Requirement already satisfied: setuptools>=11.3 in ./.envs/issue523/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Collecting cffi>=1.4.1 (from cryptography>=1.3.4; extra == "security"->requests[security])
  Using cached cffi-1.9.1.tar.gz
Requirement already satisfied: pycparser in ./.envs/issue523/lib/python3.6/site-packages (from cffi>=1.4.1->cryptography>=1.3.4; extra == "security"->requests[security])
Installing collected packages: cffi, cryptography, pyOpenSSL, requests
  Running setup.py install for cffi ... done
  Running setup.py install for cryptography ... done
Successfully installed cffi-1.9.1 cryptography-1.6 pyOpenSSL-16.2.0 requests-2.12.3
$ ~/.envs/issue523/bin/python -c "import pkg_resources; pkg_resources.require('requests[security]'); print('worked!')"
worked!

I also installed babel, but I don't have the state on my system to try to replicate the failure.

If you're running into this issue, please include steps to replicate the issue in a clean environment. Also, include the details from python -V and python -m pip --version and python -m easy_install --version (where python is the python from your env).

@zgoda

This comment has been minimized.

zgoda commented Dec 7, 2016

Okay, more info. Clean virtualenv, installed pybabel and jinja2 with deps.

$ pip list --format=columns
Package       Version 
------------- --------
Babel         2.3.4   
Jinja2        2.8     
MarkupSafe    0.23    
pip           9.0.1   
pkg-resources 0.0.0   
pytz          2016.10 
setuptools    30.2.0  
wheel         0.30.0a0
$ python -V
Python 2.7.12
$ python -m pip --version
pip 9.0.1 from /home/jazg/v/stest/local/lib/python2.7/site-packages (python 2.7)
$ python -m easy_install --version
setuptools 30.2.0 from /home/jazg/v/stest/lib/python2.7/site-packages (Python 2.7)

Possibly relevant:

$ cat babel.cfg 
[python: **.py]

[jinja2: **/templates/**.html]
extensions=jinja2.ext.autoescape,jinja2.ext.with_
encoding=utf-8

[jinja2: **/templates/**.txt]
extensions=jinja2.ext.autoescape,jinja2.ext.with_
encoding=utf-8

And the result is:

$ pybabel extract -F ./babel.cfg -k lazy_gettext -o messages.pot ./
[... skipped OK lines from python files processed...]
extracting messages from ecpapp/templates/base.html (extensions="jinja2.ext.autoescape,jinja2.ext.with_", encoding="utf-8")
Traceback (most recent call last):
  File "/home/jazg/v/stest/bin/pybabel", line 11, in <module>
    sys.exit(main())
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 911, in main
    return CommandLineInterface().run(sys.argv)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 836, in run
    return cmdinst.run()
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 454, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 156, in extract_from_dir
    dirpath=absname,
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    strip_comment_tags=strip_comment_tags
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 236, in extract_from_file
    strip_comment_tags))
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 298, in extract
    func = entry_point.load(require=True)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
@rwillmer

This comment has been minimized.

rwillmer commented Dec 12, 2016

I get this on a clean docker install based on ubuntu:xenial.

Virtualenv created with '$ virtualenv --no-site-packages '

Within the virtualenv, I have this:

$ python -V
Python 2.7.12
$ python -m pip --version
pip 9.0.1 from /usr/lib/ckan/default/local/lib/python2.7/site-packages (python 2.7)
$ python -m easy_install --version
setuptools 31.0.0 from /usr/lib/ckan/default/lib/python2.7/site-packages (Python 2.7)

And I get this:

Step 14 : RUN `pwd`/bin/paster make-config ckan /etc/ckan/default/development.ini
 ---> Running in 345382e0b9cf
/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/configuration.py:26: RuntimeWarning: Unable to load template engine entry point: 'pylonsmyghty = pylons.templating:MyghtyTemplatePlugin [myghty]': Traceback (most recent call last):
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/templating.py", line 610, in <module>
    Engine = entry_point.load()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

  import pylons.templating
@rwillmer

This comment has been minimized.

rwillmer commented Dec 12, 2016

Here's a reproducible version of the bug.
Save this as a Dockerfile:

FROM ubuntu:xenial
RUN apt-get update && apt-get -y dist-upgrade && apt-get autoremove
RUN apt-get install -y python-dev python-pip python-virtualenv git-core apt-utils libpq-dev
RUN adduser --disabled-password --gecos '' ckan
RUN mkdir -p /usr/lib/ckan/default
RUN chown ckan: /usr/lib/ckan/default
RUN mkdir -p /etc/ckan/default
RUN chown -R ckan: /etc/ckan/
USER ckan
WORKDIR /usr/lib/ckan/default
RUN virtualenv --no-site-packages `pwd`
RUN bin/pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.6.0#egg=ckan'
RUN bin/pip install -r src/ckan/requirements.txt
# ERROR:
# https://github.com/ckan/ckan/issues/3274
# https://github.com/pypa/setuptools/issues/523
RUN bin/paster make-config ckan /etc/ckan/default/development.ini

and then try to build it:

$ docker build .

You'll get the error message from my previous comment.

@rwillmer

This comment has been minimized.

rwillmer commented Dec 12, 2016

And here's a workaround which I don't understand. Downgrading and then upgrading back to the same version and it works.
If you add these two lines before the 'RUN bin/paster' line, it works fine.

RUN bin/pip install setuptools==30.4.0
RUN bin/pip install setuptools==31.0.0
@jaraco

This comment has been minimized.

Member

jaraco commented Dec 12, 2016

@rwillmer: What do you get if you invoke bin/easy_install --version prior to the workaround?

@rwillmer

This comment has been minimized.

rwillmer commented Dec 12, 2016

@jaraco

$ bin/easy_install --version 
setuptools 31.0.0 from /usr/lib/ckan/default/lib/python2.7/site-packages (Python 2.7)
@sirex

This comment has been minimized.

sirex commented Dec 16, 2016

Same here:

venv/local/lib/python2.7/site-packages/pylons/configuration.py:26: RuntimeWarning: Unable to load template engine entry point: 'pylonsmyghty = pylons.templating:MyghtyTemplatePlugin [myghty]': Traceback (most recent call last):
  File "venv/local/lib/python2.7/site-packages/pylons/templating.py", line 610, in <module>
    Engine = entry_point.load()
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "venv/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

  import pylons.templating

setuptools version:

>>> setuptools.__version__
'32.0.0'

After downgrading and upgrading setuptools as suggested by @rwillmer the error is gone.

In order to set up CKAN I was using this Ansible script:

https://github.com/sirex/ckan-ivpk-import/blob/master/ckan.yml

@xapple

This comment has been minimized.

xapple commented Jan 26, 2017

Was also trying to install CKAN, and indeed @rwillmer's fix worked.

@davidread

This comment has been minimized.

davidread commented Apr 14, 2017

Simply reinstalling the same version of setuptools solves it too. e.g.

pip install --ignore-installed setuptools==34.4.1
@zgoda

This comment has been minimized.

zgoda commented Jun 26, 2017

Still getting this on fresh virtualenv with 36.0.1:

$ python -m easy_install --version
setuptools 36.0.1 from /home/jazg/v/bwl/lib/python2.7/site-packages (Python 2.7)
@zgoda

This comment has been minimized.

zgoda commented Oct 18, 2017

Same with 36.5.0, reinstalling the same version fixes the issue:

$ pip install --ignore-installed setuptools==36.5.0

ghickman added a commit to nhsengland/iit-infrastructure that referenced this issue Feb 9, 2018

Build virtualenv as ubuntu user
Building the virtualenv as root caused us to run into [this][1]
pip/setuptools issue.  The suggested work around didn't help as the root
owned virtualenv's pip didn't actually reinstall setuptools (for reasons
I could not work out).  Installing as the ubuntu user fixes this.

[1]: pypa/setuptools#523

ghickman added a commit to nhsengland/iit-infrastructure that referenced this issue Feb 9, 2018

Build virtualenv as ubuntu user
Building the virtualenv as root caused us to run into [this][1]
pip/setuptools issue.  The suggested work around didn't help as the root
owned virtualenv's pip didn't actually reinstall setuptools (for reasons
I could not work out).  Installing as the ubuntu user fixes this.

[1]: pypa/setuptools#523
@opfertunes

This comment has been minimized.

opfertunes commented Apr 7, 2018

I had to manually downgrade setuptools to 34.00 to get around the "UndefinedEnvironmentName: 'extra' does not exist in evaluation environment." issue at runtime. (Ubuntu 16.04, python 2..7.12 virtualenv)

after running pip install -r ...

$ source ..../bin/activate
$ pip install setuptools==34.00

(34.00 is minimum required by my installed packages. The default, 39, triggers the error at runtime)

@bmjjr

This comment has been minimized.

bmjjr commented May 8, 2018

Hit this today trying to build and install from a github repo with pip install git+git://github.com/repo. Downgraded setuptools-38.5.1 to setuptools-34.00 and this resolved it so I could successfully install.

@aventurella

This comment has been minimized.

aventurella commented Oct 18, 2018

I am getting this also when installing gevent

pip==18.1
setuptools==40.4.3
gevent==1.3.7

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 349, in run
    self._warn_about_conflicts(to_install)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 475, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 108, in check_install_conflicts
    package_set, should_ignore=lambda name: name not in whitelist
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 74, in check_package_set
    missed = req.marker.evaluate()
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 301, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 226, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 208, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

However, if as @bmjjr stated, I downgrade to setuptools==34.0.0 it installs fine.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Oct 18, 2018

You'll have to give more information than that. I can't reproduce with a fresh virtualenv/venv with Python 2.7 / Python 3.7. I setuptools unvendored?

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Oct 18, 2018

Additionally, this occurs in the "check for conflicts phase" of pip's install, which means any one of the other distributions currently installed could be the problem, so it would be worth checking with pypa/pip#5842 to find out which one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment