Generic Alias attributes nor in dir()

[AllanDaemon]

BPO	41780
Nosy	@gvanrossum, @miss-islington, @isidentical, @AllanDaemon
PRs	bpo-41780: Fix __dir__ of types.GenericAlias #22262 [3.9] bpo-41780: Fix __dir__ of types.GenericAlias (GH-22262) #22264

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-15.23:38:50.719>
created_at = <Date 2020-09-14.07:06:06.931>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10']
title = 'Generic Alias attributes nor in dir()'
updated_at = <Date 2020-09-15.23:38:50.719>
user = 'https://github.com/AllanDaemon'

bugs.python.org fields:

activity = <Date 2020-09-15.23:38:50.719>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-15.23:38:50.719>
closer = 'gvanrossum'
components = ['Interpreter Core']
creation = <Date 2020-09-14.07:06:06.931>
creator = 'AllanDaemon'
dependencies = []
files = []
hgrepos = []
issue_num = 41780
keywords = ['patch']
message_count = 8.0
messages = ['376873', '376897', '376898', '376936', '376940', '376948', '376957', '376965']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'docs@python', 'miss-islington', 'BTaskaya', 'AllanDaemon']
pr_nums = ['22262', '22264']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41780'
versions = ['Python 3.9', 'Python 3.10']

[AllanDaemon]

The implementation of PEP-585 in 3.9 adds some new attributes, but they aren't listed with dir() (then, not list in autocomplete and IntelliSense).

Python 3.9.0rc1+ (heads/3.9:d7cd1164c1, Aug 25 2020, 17:27:09)

>>> li = list[int]
>>> li.__origin__
<class 'list'>
>>> getattr(li, '__origin__')
<class 'list'>
>>> '__origin__' in dir(li)
False

That applies to:

__origin__
__args__
__parameters__

It seems to be not the expected behaviour, so this bug report. If this is not the case, some piece of documentation could be interesting. Also, I couldn't find anything about Generic Alias in the documentation, only in the source code. Should this be addressed too?

[gvanrossum]

I actually have no idea where dir() gets its information. IIRC the implementation was a collaboration with Batuhan -- @batuhan, do you want to look into this? Presumably some metadata needs to be added to genericaliasobject.c.

[isidentical]

@batuhan, do you want to look into this?

Sure, will check it out!

[AllanDaemon]

I actually have no idea where dir() gets its information.

I will share what I find out already, so it may hopefully be helpful.

The dir function used is the general object.__dir__ function, defined in Objects/typeobject.c:type___dir___impl(), that just takes everything from __dict__ of the class and its bases.

But the GenericAlias is trying hard pretending to be the origin class, so when you ask for the __dict__, it gives the original class, not the GenericAlias:

>>> list[int].__dict__ == list.__dict__
True

There are also some other things that seems a bit strange, but it may just be right:

>>> list.__class__
<class 'type'>
>>> list[int].__class__
<class 'type'>

>>> type(list)
<class 'type'>
>>> type(list[int])
<class 'types.GenericAlias'>

This happens because of the tp_getattro function genericaliasobject.c:ga_getattro(), that takes the attributes from the origin except for the following names, that are taken from the GenericAlias itself:
"__origin__",
"__args__",
"__parameters__",
"__mro_entries__",
"__reduce_ex__",
"__reduce__",

So, when the dir() is getting the __dict__, it gets list.__dict__ and base (object) instead of GenericAlias. Looking the ga_getattro() calls when dir() is fired, it also gets the __class__ attribute after getting the __dict__. Adding one or another makes the dir() showing up the GenericAlias attributes, but others went missing.

An idea to solve this spefic issue could be overriding the __dir__ method,
something like this python code (written in c, of course):

def __dir__(self):
    r = super().__dir__()
    return r + attribute_names_list + method_names_list

But I wonder what else should not be forwarded to origin.

[isidentical]

Okay, so IIRC if we stop forwarding __class__ (add it as an exception to attr_exceptions) it would return us the original dir(), and also solve the inconsistency of the example you gave;

>>> list.__class__
<class 'type'>
>>> list[int].__class__
<class 'type'>

>>> type(list)
<class 'type'>
>>> type(list[int])
<class 'types.GenericAlias'>

but the problem is that, it is not 'exactly' complying with the specs at PEP-585. I am aware that we already added some extras to this list (https://www.python.org/dev/peps/pep-0585/#parameters-to-generics-are-available-at-runtime) like __mro_entries__ etc, but __class__ looks like a tricky problem. @gvanrossum any comments?

[gvanrossum]

The forwarding of __class__, and the resulting difference between type(list[int]) and list[int].__class__ are intentional.

I think overriding __dir__ along the lines suggested by Allan (but in C of course) would be the best solution.

[miss-islington]

New changeset 2e87774 by Batuhan Taskaya in branch 'master':
bpo-41780: Fix __dir__ of types.GenericAlias (GH-22262)
2e87774

[miss-islington]

New changeset 49917d5 by Miss Islington (bot) in branch '3.9':
bpo-41780: Fix __dir__ of types.GenericAlias (GH-22262)
49917d5