PEP 563: Should the behavior change for yield/yield from's

[isidentical]

BPO	42725
Nosy	@gvanrossum, @larryhastings, @markshannon, @serhiy-storchaka, @gvanrossum, @lysnikolaou, @pablogsal, @miss-islington, @isidentical
PRs	bpo-42725: Render annotations effectless on symbol table with PEP 563 #25583 bpo-42725: mention the changes on what's new #25974 [3.10] bpo-42725: mention the changes on what's new (GH-25974) #25988

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-08.13:03:31.147>
created_at = <Date 2020-12-23.13:28:51.066>
labels = []
title = "PEP 563: Should the behavior change for yield/yield from's"
updated_at = <Date 2021-05-08.13:03:31.146>
user = 'https://github.com/isidentical'

bugs.python.org fields:

activity = <Date 2021-05-08.13:03:31.146>
actor = 'BTaskaya'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-08.13:03:31.147>
closer = 'BTaskaya'
components = []
creation = <Date 2020-12-23.13:28:51.066>
creator = 'BTaskaya'
dependencies = []
files = []
hgrepos = []
issue_num = 42725
keywords = ['patch']
message_count = 34.0
messages = ['383647', '383655', '383659', '383660', '383661', '383662', '383686', '383693', '383730', '383812', '384483', '384508', '384580', '384589', '384602', '384617', '384638', '385203', '385210', '385217', '385223', '391723', '391747', '391748', '391750', '391754', '391764', '391780', '391782', '391783', '392777', '392803', '392831', '393252']
nosy_count = 9.0
nosy_names = ['gvanrossum', 'larry', 'Mark.Shannon', 'serhiy.storchaka', 'Guido.van.Rossum', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'BTaskaya']
pr_nums = ['25583', '25974', '25988']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42725'
versions = []

[isidentical]

Since the annotations are processed just like all other expressions in the symbol table, the generated entries for functions etc. This would result with

def foo():
    for number in range(5):
        foo: (yield number)
    return number

foo()

returning a generator / coroutine (depending on yield/yield from/await usage). Is this something we want to keep or maybe tweak the symbol table generator to not to handle annotations (since there are also more subtle issues regarding analysis of cell / free vars).

[gvanrossum]

Ouch. I think we should generate a SyntaxError if yield [from] is used in an annotation.

[isidentical]

This is another side effect of processing annotations (at the symbol table construction stage) (and I would assume there are a few more cases like this);

def foo():
    outer_var = 1

    def bar():
        inner_var: outer_var = T
    
    return bar

inner = foo()
print(inner.__closure__)

In theory, there shouldn't be any cells / references to the variables from outer scope, but since we process the entry for the annotation and record outer_var as a free var it is listed here.

[gvanrossum]

The difference is that that just causes a slight inefficiency, while processing 'yield' makes the function a generator. So I don't feel as strongly about that.

[isidentical]

So I don't feel as strongly about that.

Just to note, since I believe the solution for all this might be the same (not processing annotations at all, since they will be compiled to strings in the later stage). If we go down on that route, it will be simpler but we won't be able to raise SyntaxError's for "a: (yield)" / "b: (await/yield from x)".

By the way, this is what happens if you try to use get_type_hints on a function where an argument is (yield):

>>> typing.get_type_hints(a)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/typing.py", line 537, in __init__
    code = compile(arg, '<string>', 'eval')
  File "<string>", line 1
SyntaxError: 'yield' outside function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/typing.py", line 1490, in get_type_hints
    value = ForwardRef(value)
  File "/usr/local/lib/python3.10/typing.py", line 539, in __init__
    raise SyntaxError(f"Forward reference must be an expression -- got {arg!r}")
SyntaxError: Forward reference must be an expression -- got '(yield)'

[gvanrossum]

If we simply ignored yield in the annotation, wouldn't we have the problem that

def f(a: (yield)): pass

silently changes from being a generator (in 3.9) to not being a generator (in 3.10)? That would be bad. I'd rather make this an error still. (But for nonlocals, not processing sounds fine.)

[lysnikolaou]

I concur with Guido. I feel that making this an error is the best alternative we have.

[gvanrossum]

Okay, let's do it. It should probably a post-parse check (before we invoke the bytecode compiler but after we have the AST in hand).

[isidentical]

I have a patch ready to go, but I'd prefer to block this issue until bpo-42737 is resolved/decided.

[isidentical]

One thing to note here, currently Pablo and I are trying to bring annotation unparsing from the compiler to the parser in bpo-41967. If we do so, the annotations won't cause any side effects on the symbol table generation.

[markshannon]

I've also opened bpo-42837 which is about fixing the symbol table, so that it is correct w.r.t. to current behavior.

I'd like to fix it ASAP as the compiler should be able to rely on the symbol table being correct.

Of course, once we have decided what the behavior should be, then it may need to be updated again.

I'm inclined to agree that 'yield' in an annotation should probably be a syntax error, but I haven't put much thought into to.

[gvanrossum]

Yield in an annotation should be a syntax error.

[markshannon]

What's the process for making a decision on whether to make 'yield' in an annotation a syntax error?

As a language change it should have a PEP, IMO.
The PEP will be short, and shouldn't need a long-winded acceptance process.
I just think that a PEP is more visible to the community than the bug tracker.

I'm happy to write the PEP.

[serhiy-storchaka]

Should not "await" and "async for" (in comprehesions) and ":=" be forbidden too?

[gvanrossum]

I wouldn’t have thought you’d need a PEP for this but if you want to write one that sounds like the right thing.

Re: async and walrus: I think those are different, their presence doesn’t affect the meaning of the function like yield. We can’t hope to prevent side effects syntactically.