Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

__init_subclass__ causes TypeError when used with standard library metaclasses (such as ABCMeta) #73767

Closed
NateSoares mannequin opened this issue Feb 16, 2017 · 9 comments
Labels
3.7 (EOL) end of life type-bug An unexpected behavior, bug, or error

Comments

@NateSoares
Copy link
Mannequin

NateSoares mannequin commented Feb 16, 2017

BPO 29581
Nosy @ncoghlan, @ambv, @tecki, @ilevkivskyi, @Mariatta
PRs
  • bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ #527
  • [3.6] bpo-29581: Make __init_subclass__ work on ABCMeta type classes #1282
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-06-16.08:17:14.953>
    created_at = <Date 2017-02-16.18:54:07.670>
    labels = ['type-bug', '3.7']
    title = '__init_subclass__ causes TypeError when used with standard library metaclasses (such as ABCMeta)'
    updated_at = <Date 2017-06-16.08:17:14.951>
    user = 'https://bugs.python.org/NateSoares'

    bugs.python.org fields:

    activity = <Date 2017-06-16.08:17:14.951>
    actor = 'ncoghlan'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-06-16.08:17:14.953>
    closer = 'ncoghlan'
    components = []
    creation = <Date 2017-02-16.18:54:07.670>
    creator = 'Nate Soares'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 29581
    keywords = []
    message_count = 9.0
    messages = ['287966', '288119', '288255', '288450', '288451', '288453', '290179', '295312', '296165']
    nosy_count = 7.0
    nosy_names = ['ncoghlan', 'lukasz.langa', 'Martin.Teichmann', 'levkivskyi', 'Kevin Shweh', 'Mariatta', 'Nate Soares']
    pr_nums = ['527', '1282']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue29581'
    versions = ['Python 3.6', 'Python 3.7']

    @NateSoares
    Copy link
    Mannequin Author

    NateSoares mannequin commented Feb 16, 2017

    I believe I've found a bug (or, at least, critical shortcoming) in the way that python 3.6's __init_subclass__ interacts with abc.ABCMeta (and, presumably, most other metaclasses in the standard library). In short, if a class subclasses both an abstract class and a class-that-uses-init_subclass, and the __init_subclass__ uses keyword arguments, then this will often lead to TypeErrors (because the metaclass gets confused by the keyword arguments to __new__ that were meant for __init_subclass__).

    Here's an example of the failure. This code:

    from abc import ABCMeta
    class Initifier:
        def __init_subclass__(cls, x=None, **kwargs):
            super().__init_subclass__(**kwargs)
            print('got x', x)
    
    class Abstracted(metaclass=ABCMeta):
        pass
    
    class Thingy(Abstracted, Initifier, x=1):
        pass
    
    thingy = Thingy()

    raises this TypeError when run:

    Traceback (most recent call last):
      File "<filename>", line 10, in <module>
        class Thingy(Abstracted, Initifier, x=1):
    TypeError: __new__() got an unexpected keyword argument 'x'

    See http://stackoverflow.com/questions/42281697/typeerror-when-combining-abcmeta-with-init-subclass-in-python-3-6 for further discussion.

    @NateSoares NateSoares mannequin added the type-bug An unexpected behavior, bug, or error label Feb 16, 2017
    @ncoghlan
    Copy link
    Contributor

    This is going to be the case anytime an attempt is made to combine parent classes with incompatible constructor signatures. "type" is unusual in that it goes to great lengths to present an adaptive signature that aligns with whatever the class definition does.

    The main relevant trick is to filter the extra arguments out from those passed to metaclasses such that only init_subclass sees them:

    ================

    def ignore_extra_args(base):
        base_meta = type(base)
        class _FilteredMeta(base_meta):
            def __new__(*args, **kwds):
                return base_meta.__new__(*args)
            def __init__(*args, **kwds):
                return base_meta.__init__(*args)
        class _Filtered(base, metaclass=_FilteredMeta):
            pass
        return _Filtered
    
    class InitX():
        def __init_subclass__(cls, x=None):
            print('x')
    
    from abc import ABCMeta
    class Abstract(metaclass=ABCMeta):
        pass
    
    class AbstractWithInit(ignore_extra_args(Abstract), InitX, x=1):
        pass
    
    AbstractWithInit()

    ================

    If folks were to iterate on "ignore_extra_args" variants outside the standard library with 3.6, then it would be something we could sensibly standardise for 3.7.

    @KevinShweh
    Copy link
    Mannequin

    KevinShweh mannequin commented Feb 21, 2017

    Doesn't that ignore_extra_args thing prevent InitX.__init_subclass__ from receiving the x argument it wanted? It doesn't seem like a solution.

    @ncoghlan
    Copy link
    Contributor

    No, the filtering is only applied to the __new__ and __init__ calls on the metaclass, not to the __init_subclass__ call.

    Showing the last part of an interactive session where I ran the above commands:

    ===========

    >>> class AbstractWithInit(ignore_extra_args(Abstract), InitX, x=1):
    ...     pass
    ... 
    x
    >>> AbstractWithInit()
    <__main__.AbstractWithInit object at 0x7f9086694a58>
    >>>

    ===========

    @ncoghlan
    Copy link
    Contributor

    Oops, and now I see the typo in the example code, it seems you're right. I was misremembering an earlier more decorator-like variant of the design where we didn't rely on passing the __init_subclass__ arguments through the metaclass constructor.

    @ncoghlan
    Copy link
    Contributor

    Note that the publisher side workaround for this is relatively straightforward: __init_subclass__ implementations that want to be compatible with arbitrary metaclasses will need to take any additional parameters as class attributes (which they may delete), rather than as class header keyword arguments.

    For 3.7 though, it probably makes sense to update abc.ABCMeta to pass along arbitrary keyword arguments based on the same rationale as used in PEP-487 to justify making this change for type itself: by default, type.__init_subclass__ will still complain about it, but if someone overrides __init_subclass__ to accept additional keyword arguments, doing so will just work.

    @ncoghlan ncoghlan added the 3.7 (EOL) end of life label Feb 23, 2017
    @ambv
    Copy link
    Contributor

    ambv commented Mar 24, 2017

    New changeset bd583ef by Łukasz Langa (Nate) in branch 'master':
    bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ (#527)
    bd583ef

    @Mariatta
    Copy link
    Member

    Mariatta commented Jun 7, 2017

    New changeset 6fb12b5 by Mariatta (Nate) in branch '3.6':
    bpo-29581: bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ (GH-527) (GH-1282)
    6fb12b5

    @ncoghlan
    Copy link
    Contributor

    Thanks for the work on resolving this, all!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants