bpo-44747: Refactor usage of sys._getframe at typing module

[uriyyo]

https://bugs.python.org/issue44747

[uriyyo]

@Fidget-Spinner Could you please review this PR?

[Fidget-Spinner]

Thanks, this mostly looks good. Could you please change your PR title to Refactor usage... instead of Reduce usage...? I thought you'd found some clever hack to avoid sys._getframe outright at first 😉 .

BTW, after ae0a2b7, avoiding sys._getframe could yield us very good performance gains. So there's some incentive to that, though I don't know how we can get rid of it.

[Fidget-Spinner]

I'm confused. Why don't we need this anymore? I thought the new default is __main__ which is different?

[uriyyo]

Looks like we should do it like it did at other classes:

def_mod = _callee()
if def_mod != 'typing':
    self.__module__ = def_mod

This line was written by me and it can lead to pickling errors cause in case if name is not present it will set module name to typing and typing module can not contains this NewType.

What is your opinion regarding that?

[Fidget-Spinner]

Wow interesting! Are you able to to reproduce this with a small peice of code? If you can, we should backport this specific change (and you should open a separate PR for this, tied to the pickling/serialization issue). We can't backport the current PR entirely because it is refactoring, not bugfix.

[uriyyo]

Code to snippet to reproduce this issue:

import pickle
from typing import NewType

del __name__

MyNewType = NewType("MyNewType", int)

pickle.dumps(MyNewType)

Traceback (most recent call last):
  File "...", line 9, in <module>
    pickle.dumps(MyNewType)
    ^^^^^^^^^^^^^^^^^^^^^^^
_pickle.PicklingError: Can't pickle typing.MyNewType: attribute lookup MyNewType on typing failed

I am not sure that it's okay to delete __name__ attribute of module)

[Fidget-Spinner]

Thanks. Please create a separate PR just for this and link to https://bugs.python.org/issue44353.

[uriyyo]

New issue has been created.

link to issue - https://bugs.python.org/issue44761
lint to PR - #27406

[Fidget-Spinner]

Hmm, also I can't verify this from the devguide, but I had the impression that refactoring can usually skip news if it has zero effect on the end user. I'll wait and see what others say.

[Fidget-Spinner]

LGTM. Lets wait for your other PR to land first. BTW minor nit and feel free to ignore it: (see #27387 (comment))

[ambv]

I find this goes somewhat outside of an acceptable refactor for backporting to 3.10 and 3.9 due to changes to _allow_reckless_class_checks.

[uriyyo]

@Fidget-Spinner @ambv How do you think typing._callee implementation should look like? Is current implementation to complex? I want to hear yours opinion regarding that to make implementation better.

[ambv]

The implementation of _callee() is close to what we need. Its contract is essentially to return the caller without ever raising exceptions. I would simplify:

• if _getframe() is supported, then you will never get None back because default is '__main__' and __name__ is only ever explicitly set to None during interpreter shutdown;
• therefore, we can remove the unsupported= kwarg and just return None in this case.

In this case usage of _callee() could use the result as long as it is not None.

[ambv]

Oh, by the way, it just appeared to me. The function is called wrong.

callee - n. The person who is called by the caller (on the telephone)

The function should be called _caller, not _callee.

[uriyyo]

if _getframe() is supported, then you will never get None back because default is 'main' and name is only ever explicitly set to None during interpreter shutdown;

Looks like we can explicitly set __name__ to None and in such case _caller will return None because __name__ will be at module namespace.

Code to show this behavior:

ns = {}

exec("""
import typing

__name__ = None

T = typing.TypeVar("T")
P = typing.ParamSpec("P")
NT = typing.NewType("NT", int)
""", ns)

for o in (ns["T"], ns["P"], ns["NT"]):
    print(o, o.__module__)

~T None
~P None
None.NT None

I don't know if it's valid case when someone explicitly set __name__ value. Should we cover this case or can we skip it?

[ambv]

If the user explicitly sets __name__ to an invalid value of None, types declared in such module will have __module__ set to None as well. Other weird things will happen too like the inability to use logging filtering, and so on.

The only time __name__ is set to None is during interpreter shutdown. We don't care about this use case since by that time no new typing.* objects are being created.

[bedevere-bot]

@ambv: Please replace # with GH- in the commit message next time. Thanks!