New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools + pbr broken on Jython 2.7.0 #1024

Closed
LordGaav opened this Issue Apr 26, 2017 · 7 comments

Comments

Projects
None yet
2 participants
@LordGaav
Contributor

LordGaav commented Apr 26, 2017

I'm running into an issue using a combination of setuptools and pbr on Jython, that produces this error:

Traceback (most recent call last):
  File "setup.py", line 24, in <module>
    setup(
  File "/home/chaos/jython2.7.0/Lib/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/home/chaos/jython2.7.0/Lib/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "/home/chaos/workspace/omniconf/vendor/Lib/site-packages/setuptools/dist.py", line 320, in __init__
    _Distribution.__init__(self, attrs)
  File "/home/chaos/jython2.7.0/Lib/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/home/chaos/workspace/omniconf/vendor/Lib/site-packages/setuptools/dist.py", line 387, in finalize_options
    ep.load()(self, ep.name, value)
  File "/home/chaos/workspace/omniconf/.eggs/pbr-3.0.0-py2.7.egg/pbr/core.py", line 98, in pbr
    _monkeypatch_distribution()
  File "/home/chaos/workspace/omniconf/.eggs/pbr-3.0.0-py2.7.egg/pbr/core.py", line 62, in _monkeypatch_distribution
    core.Distribution = dist._get_unpatched(core.Distribution)
  File "/home/chaos/workspace/omniconf/vendor/Lib/site-packages/setuptools/dist.py", line 29, in _get_unpatched
    return get_unpatched(cls)
  File "/home/chaos/workspace/omniconf/vendor/Lib/site-packages/setuptools/monkey.py", line 30, in get_unpatched
    return lookup(item)
  File "/home/chaos/workspace/omniconf/vendor/Lib/site-packages/setuptools/monkey.py", line 44, in get_unpatched_class
    base = next(external_bases)
StopIteration

All versions >= 32.3.0 are affected, which leads me to believe that this change introduced the bug.

Steps to reproduce:

  1. Install Jython 2.7.0 (see https://github.com/LordGaav/omniconf/blob/develop/.travis.yml for steps)
  2. Update and install pip, setuptools and virtualenv in Jython: /path/to/jython/bin/pip install -U pip setuptools virtualenv
  3. Checkout my fork of omniconf: https://github.com/LordGaav/omniconf and checkout setuptools+pbr+jython-bug .
  4. Create a Jython virtualenv: /path/to/jython/bin/virtualenv env and enter it.
  5. python setup.py install will produce the error.
  6. Do pip install setuptools==32.0.0 .
  7. python setup.py install works as expected.

LordGaav added a commit to LordGaav/omniconf that referenced this issue Apr 26, 2017

@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco Apr 26, 2017

Member

That change was made intentionally to address #889. Can you debug in the Jython environment and determine where the behavior goes awry? Is it that inspect.getmro behaves differently on Jython? What does the result of inspect.getmro(cls) look like in that context? What does cls.__bases__ look like?

Member

jaraco commented Apr 26, 2017

That change was made intentionally to address #889. Can you debug in the Jython environment and determine where the behavior goes awry? Is it that inspect.getmro behaves differently on Jython? What does the result of inspect.getmro(cls) look like in that context? What does cls.__bases__ look like?

@LordGaav

This comment has been minimized.

Show comment
Hide comment
@LordGaav

LordGaav Apr 26, 2017

Contributor

Given a virtualenv with Jython 2.7.0 and these dependencies:

$ pip freeze --all
appdirs==1.4.3
packaging==16.8
pbr==3.0.0
pip==9.0.1
pyparsing==2.2.0
setuptools==35.0.1
six==1.10.0
wheel==0.29.0

I changed the method to this:

def get_unpatched_class(cls):
    """Protect against re-patching the distutils if reloaded

    Also ensures that no other distutils extension monkeypatched the distutils
    first.
    """
    print("MRO")
    print(inspect.getmro(cls))
    print("Bases")
    print(cls.__bases__)
    external_bases = (
        cls
        for cls in inspect.getmro(cls)
        if not cls.__module__.startswith('setuptools')
    )
    print("external_bases")
    print(external_bases)
    print("---")
    base = next(external_bases)
    if not base.__module__.startswith('distutils'):
        msg = "distutils has already been patched by %r" % cls
        raise AssertionError(msg)
    return base

Which produced this output when calling python setup.py install:

$ python setup.py install
MRO
(<class distutils.extension.Extension at 0x19a>,)
Bases
()
external_bases
<generator object at 0x19b>
---
MRO
(<class distutils.dist.Distribution at 0x19d>,)
Bases
()
external_bases
<generator object at 0x19e>
---
MRO
(<class distutils.cmd.Command at 0x19f>,)
Bases
()
external_bases
<generator object at 0x1a0>
---
MRO
(<class setuptools.dist.Distribution at 0x1a1>, <class setuptools.py36compat.Distribution_parse_config_files at 0x1a2>)
Bases
(<class setuptools.py36compat.Distribution_parse_config_files at 0x1a2>, <class distutils.dist.Distribution at 0x19d>)
external_bases
<generator object at 0x1a3>
---

When doing the same on Python 3.4, this is the output:

$ python setup.py install                                                                                                                                                                          
MRO
(<class 'distutils.extension.Extension'>, <class 'object'>)
Bases
(<class 'object'>,)
external_bases
<generator object <genexpr> at 0x7ff9b616b7e0>
---
MRO
(<class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'object'>,)
external_bases
<generator object <genexpr> at 0x7ff9b5a2bf30>
---
MRO
(<class 'distutils.cmd.Command'>, <class 'object'>)
Bases
(<class 'object'>,)
external_bases
<generator object <genexpr> at 0x7ff9b5a2bfc0>
---
MRO
(<class 'setuptools.dist.Distribution'>, <class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>)
external_bases
<generator object <genexpr> at 0x7ff9b5a084c8>
---
Contributor

LordGaav commented Apr 26, 2017

Given a virtualenv with Jython 2.7.0 and these dependencies:

$ pip freeze --all
appdirs==1.4.3
packaging==16.8
pbr==3.0.0
pip==9.0.1
pyparsing==2.2.0
setuptools==35.0.1
six==1.10.0
wheel==0.29.0

I changed the method to this:

def get_unpatched_class(cls):
    """Protect against re-patching the distutils if reloaded

    Also ensures that no other distutils extension monkeypatched the distutils
    first.
    """
    print("MRO")
    print(inspect.getmro(cls))
    print("Bases")
    print(cls.__bases__)
    external_bases = (
        cls
        for cls in inspect.getmro(cls)
        if not cls.__module__.startswith('setuptools')
    )
    print("external_bases")
    print(external_bases)
    print("---")
    base = next(external_bases)
    if not base.__module__.startswith('distutils'):
        msg = "distutils has already been patched by %r" % cls
        raise AssertionError(msg)
    return base

Which produced this output when calling python setup.py install:

$ python setup.py install
MRO
(<class distutils.extension.Extension at 0x19a>,)
Bases
()
external_bases
<generator object at 0x19b>
---
MRO
(<class distutils.dist.Distribution at 0x19d>,)
Bases
()
external_bases
<generator object at 0x19e>
---
MRO
(<class distutils.cmd.Command at 0x19f>,)
Bases
()
external_bases
<generator object at 0x1a0>
---
MRO
(<class setuptools.dist.Distribution at 0x1a1>, <class setuptools.py36compat.Distribution_parse_config_files at 0x1a2>)
Bases
(<class setuptools.py36compat.Distribution_parse_config_files at 0x1a2>, <class distutils.dist.Distribution at 0x19d>)
external_bases
<generator object at 0x1a3>
---

When doing the same on Python 3.4, this is the output:

$ python setup.py install                                                                                                                                                                          
MRO
(<class 'distutils.extension.Extension'>, <class 'object'>)
Bases
(<class 'object'>,)
external_bases
<generator object <genexpr> at 0x7ff9b616b7e0>
---
MRO
(<class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'object'>,)
external_bases
<generator object <genexpr> at 0x7ff9b5a2bf30>
---
MRO
(<class 'distutils.cmd.Command'>, <class 'object'>)
Bases
(<class 'object'>,)
external_bases
<generator object <genexpr> at 0x7ff9b5a2bfc0>
---
MRO
(<class 'setuptools.dist.Distribution'>, <class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>)
external_bases
<generator object <genexpr> at 0x7ff9b5a084c8>
---
@LordGaav

This comment has been minimized.

Show comment
Hide comment
@LordGaav

LordGaav Apr 26, 2017

Contributor

Some more data, when checking how Jython handles cls.__module__, I changed the function to look like this:

def get_unpatched_class(cls):
    """Protect against re-patching the distutils if reloaded

    Also ensures that no other distutils extension monkeypatched the distutils
    first.
    """
    print("MRO")
    print(inspect.getmro(cls))
    print("Bases")
    print(cls.__bases__)
    external_bases = []
    for cls in inspect.getmro(cls):
        print(cls.__module__)
        if not cls.__module__.startswith('setuptools'):
            external_bases.append(cls)
    print("external_bases")
    print(external_bases)
    print("---")
    base = next(iter(external_bases))
    if not base.__module__.startswith('distutils'):
        msg = "distutils has already been patched by %r" % cls
        raise AssertionError(msg)
    return base

Which produces on Jython:

$ python setup.py install
MRO
(<class distutils.extension.Extension at 0x19a>,)
Bases
()
distutils.extension
external_bases
[<class distutils.extension.Extension at 0x19a>]
---
MRO
(<class distutils.dist.Distribution at 0x19c>,)
Bases
()
distutils.dist
external_bases
[<class distutils.dist.Distribution at 0x19c>]
---
MRO
(<class distutils.cmd.Command at 0x19d>,)
Bases
()
distutils.cmd
external_bases
[<class distutils.cmd.Command at 0x19d>]
---
MRO
(<class setuptools.dist.Distribution at 0x19e>, <class setuptools.py36compat.Distribution_parse_config_files at 0x19f>)
Bases
(<class setuptools.py36compat.Distribution_parse_config_files at 0x19f>, <class distutils.dist.Distribution at 0x19c>)
setuptools.dist
setuptools.py36compat
external_bases
[]
---

And on Python 3.4:

$ python setup.py install
MRO
(<class 'distutils.extension.Extension'>, <class 'object'>)
Bases
(<class 'object'>,)
distutils.extension
builtins
external_bases
[<class 'distutils.extension.Extension'>, <class 'object'>]
---
MRO
(<class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'object'>,)
distutils.dist
builtins
external_bases
[<class 'distutils.dist.Distribution'>, <class 'object'>]
---
MRO
(<class 'distutils.cmd.Command'>, <class 'object'>)
Bases
(<class 'object'>,)
distutils.cmd
builtins
external_bases
[<class 'distutils.cmd.Command'>, <class 'object'>]
---
MRO
(<class 'setuptools.dist.Distribution'>, <class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>)
setuptools.dist
setuptools.py36compat
distutils.dist
builtins
external_bases
[<class 'distutils.dist.Distribution'>, <class 'object'>]
---

It does seem that inspect.getmro behaves differently on Jython than on Python 3.4. distutils.dist.Distribution is missing, which I assume is the base of setuptools.dist.Distribution .

Contributor

LordGaav commented Apr 26, 2017

Some more data, when checking how Jython handles cls.__module__, I changed the function to look like this:

def get_unpatched_class(cls):
    """Protect against re-patching the distutils if reloaded

    Also ensures that no other distutils extension monkeypatched the distutils
    first.
    """
    print("MRO")
    print(inspect.getmro(cls))
    print("Bases")
    print(cls.__bases__)
    external_bases = []
    for cls in inspect.getmro(cls):
        print(cls.__module__)
        if not cls.__module__.startswith('setuptools'):
            external_bases.append(cls)
    print("external_bases")
    print(external_bases)
    print("---")
    base = next(iter(external_bases))
    if not base.__module__.startswith('distutils'):
        msg = "distutils has already been patched by %r" % cls
        raise AssertionError(msg)
    return base

Which produces on Jython:

$ python setup.py install
MRO
(<class distutils.extension.Extension at 0x19a>,)
Bases
()
distutils.extension
external_bases
[<class distutils.extension.Extension at 0x19a>]
---
MRO
(<class distutils.dist.Distribution at 0x19c>,)
Bases
()
distutils.dist
external_bases
[<class distutils.dist.Distribution at 0x19c>]
---
MRO
(<class distutils.cmd.Command at 0x19d>,)
Bases
()
distutils.cmd
external_bases
[<class distutils.cmd.Command at 0x19d>]
---
MRO
(<class setuptools.dist.Distribution at 0x19e>, <class setuptools.py36compat.Distribution_parse_config_files at 0x19f>)
Bases
(<class setuptools.py36compat.Distribution_parse_config_files at 0x19f>, <class distutils.dist.Distribution at 0x19c>)
setuptools.dist
setuptools.py36compat
external_bases
[]
---

And on Python 3.4:

$ python setup.py install
MRO
(<class 'distutils.extension.Extension'>, <class 'object'>)
Bases
(<class 'object'>,)
distutils.extension
builtins
external_bases
[<class 'distutils.extension.Extension'>, <class 'object'>]
---
MRO
(<class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'object'>,)
distutils.dist
builtins
external_bases
[<class 'distutils.dist.Distribution'>, <class 'object'>]
---
MRO
(<class 'distutils.cmd.Command'>, <class 'object'>)
Bases
(<class 'object'>,)
distutils.cmd
builtins
external_bases
[<class 'distutils.cmd.Command'>, <class 'object'>]
---
MRO
(<class 'setuptools.dist.Distribution'>, <class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>, <class 'object'>)
Bases
(<class 'setuptools.py36compat.Distribution_parse_config_files'>, <class 'distutils.dist.Distribution'>)
setuptools.dist
setuptools.py36compat
distutils.dist
builtins
external_bases
[<class 'distutils.dist.Distribution'>, <class 'object'>]
---

It does seem that inspect.getmro behaves differently on Jython than on Python 3.4. distutils.dist.Distribution is missing, which I assume is the base of setuptools.dist.Distribution .

@LordGaav

This comment has been minimized.

Show comment
Hide comment
@LordGaav

LordGaav Apr 26, 2017

Contributor

Changing the method to this seems to fix the issue by sidestepping inspect.getmro:

def get_unpatched_class(cls):
    """Protect against re-patching the distutils if reloaded
    Also ensures that no other distutils extension monkeypatched the distutils
    first.
    """
    external_bases = (
        cls
        for cls in (cls,) + cls.__bases__
        if not cls.__module__.startswith('setuptools')
    )
    base = next(external_bases)
    if not base.__module__.startswith('distutils'):
        msg = "distutils has already been patched by %r" % cls
        raise AssertionError(msg)
    return base

I'm not sure if this introduces any other issues. inspect.getmro seems to just return a list of all base classes (including the class itself), sorted by the MRO. cls.__bases__ also seems to do this, except it doesn't include the class itself.

If this is an acceptable solution, I can prepare a pull request for you.

Contributor

LordGaav commented Apr 26, 2017

Changing the method to this seems to fix the issue by sidestepping inspect.getmro:

def get_unpatched_class(cls):
    """Protect against re-patching the distutils if reloaded
    Also ensures that no other distutils extension monkeypatched the distutils
    first.
    """
    external_bases = (
        cls
        for cls in (cls,) + cls.__bases__
        if not cls.__module__.startswith('setuptools')
    )
    base = next(external_bases)
    if not base.__module__.startswith('distutils'):
        msg = "distutils has already been patched by %r" % cls
        raise AssertionError(msg)
    return base

I'm not sure if this introduces any other issues. inspect.getmro seems to just return a list of all base classes (including the class itself), sorted by the MRO. cls.__bases__ also seems to do this, except it doesn't include the class itself.

If this is an acceptable solution, I can prepare a pull request for you.

@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco Apr 27, 2017

Member

I don't understand why inspect.getmro isn't returning the full mro on Jython. I've tried replicating what you're seeing with a simple example and I'm unable to do so. Does the issue you have only occur when you're using pbr or can you make it happen with setuptools alone? I'd prefer to know more about the underlying cause.

I'm disinclined to accept the workaround you've provided because __bases__ and getmro return different things more than just the leading class. getmro also returns bases of bases, or at least it should. If we are to move forward with a workaround, I'd want the workaround to be implemented as a standalone adapter with documented purpose and only invoked on Jython, relying on the preferred implementation by default.

Member

jaraco commented Apr 27, 2017

I don't understand why inspect.getmro isn't returning the full mro on Jython. I've tried replicating what you're seeing with a simple example and I'm unable to do so. Does the issue you have only occur when you're using pbr or can you make it happen with setuptools alone? I'd prefer to know more about the underlying cause.

I'm disinclined to accept the workaround you've provided because __bases__ and getmro return different things more than just the leading class. getmro also returns bases of bases, or at least it should. If we are to move forward with a workaround, I'd want the workaround to be implemented as a standalone adapter with documented purpose and only invoked on Jython, relying on the preferred implementation by default.

@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco Apr 27, 2017

Member

I've figured it out - it's a bug/limitation of Jython. The issue appears to be that because both distutils.dist.Distribution and setuptools.dist.Distribution are called Distribution, getmro only returns the first. You can see the same running the following on Jython:

class OtherNamespace:
	class Y:
		pass

class Y:
 pass

class Z(OtherNamespace.Y, Y):
 pass

import inspect

print(inspect.getmro(Z))

It outputs

(<class __main__.Z at 0x2>, <class __main__.Y at 0x3>)

Run it on CPython and you'll see multiple Y classes.

(<class __main__.Z at 0x10100cbb0>, <class __main__.Y at 0x10100ca78>, <class __main__.Y at 0x10100cb48>)
Member

jaraco commented Apr 27, 2017

I've figured it out - it's a bug/limitation of Jython. The issue appears to be that because both distutils.dist.Distribution and setuptools.dist.Distribution are called Distribution, getmro only returns the first. You can see the same running the following on Jython:

class OtherNamespace:
	class Y:
		pass

class Y:
 pass

class Z(OtherNamespace.Y, Y):
 pass

import inspect

print(inspect.getmro(Z))

It outputs

(<class __main__.Z at 0x2>, <class __main__.Y at 0x3>)

Run it on CPython and you'll see multiple Y classes.

(<class __main__.Z at 0x10100cbb0>, <class __main__.Y at 0x10100ca78>, <class __main__.Y at 0x10100cb48>)
@jaraco

This comment has been minimized.

Show comment
Hide comment
@jaraco

jaraco Apr 27, 2017

Member

I've [filed an issue upstream]. At this point, I'll welcome a patch per my guidelines above.

Member

jaraco commented Apr 27, 2017

I've [filed an issue upstream]. At this point, I'll welcome a patch per my guidelines above.

LordGaav added a commit to LordGaav/setuptools that referenced this issue Apr 27, 2017

Use a different method to lookup base classes on Jython
Jython seems to implement inspect.getmro differently, which causes
any classes with the same name as a class lower in the MRO not to
be returned.

This patch offloads the MRO lookup to a separate function, which
implements different logic for Jython only.

Ref #1024

@jaraco jaraco closed this in 498fec2 Apr 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment