-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
yield expression inside generator expression does nothing #54753
Comments
Simple coroutine with for loop works: >>> def pack_a():
while True:
L = []
for i in range(2):
L.append((yield))
print(L)
>>> pa = pack_a()
>>> next(pa)
>>> pa.send(1)
>>> pa.send(2)
[1, 2]
>>> If using list comprehension (generator expression), it fails: >>> def pack_b():
while True:
L = [(yield) for i in range(2)]
print(L)
>>> pb = pack_b()
<endless loop here> I understand what's going on here - generator expression is converted to nested function and there's no way to either stop the execution inside nested function (since it's not started yet!) or send() a value to its yield expression. Still I think this behavior is a bug and needs fixed.
|
Hmm, what an interesting and unexpected side-effect of the efforts to hide the loop induction variable. |
While the behavior is confusing, I don't think yield inside comprehensions should be disallowed. Rather, the fact that comprehensions have their own scope should be stated clearer. |
I think I can probably fix it, but it's debatable whether it should be done, since it'd make list comps more of "quasi" functions. |
This discussion should probably be moved to python-dev. With tools like Twisted's inlineDefer or the Monocle package, there is a growing need to be able to use yield in complex expressions. Yet, that goes against the trend toward making lists comps more like genexps and less like sugar for a simple for-loop accumulator. Guido, do you have any thoughts on the subject? Mark it a documentation issue or try out Benjamin's fix? |
I think it is definitely wrong the way it works in 3.x. (Especially since it works as expected in 2.x.) I agree with Inyeol's preference of fixes: (1) make it work properly for listcomps as well as genexps, (2) if that's not possible, forbid yield in a genexp or listcomp. Note that even though yield in a genexp could be considered as having a well-defined meaning, that meaning is not useful and I would consider it as merely a coincidence of the specification, not an intentional effect. So I would be fine changing its meaning. (My assumption is that since it is not useful there is -- almost -- no code depending on that meaning.) |
PS. Wasn't there a similar issue with something inside a genexp that raises StopIteration? Did we ever solve that? |
Isn't this the same issue as bpo-3267? |
Yes it is, but I was never asked about it back then. |
bpo-3267 did not expose endless loop possibility and was closed as won't fix. |
FWIW, the "endless loop possibility" is not of issue here, and is simply an artifact of the specific generator function the OP uses. |
I would like to add that since the introduction of asyncio module that heavily uses "yield from" syntax, binding of yield inside comprehensions/generator expressions could lead to unexpected results/confusing behavior. See for example this question on SO: http://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin |
Is the fact that 'await' produces a syntax error in this context the same bug or a new one? |
What kind of SyntaxError? await outside an async function is prohibited, bare await is also prohibited. |
>>> async def foo():
... bar = [await x for x in range(10)]
File "<input>", line 2
SyntaxError: 'await' expressions in comprehensions are not supported |
Python 3.5 does not support this, you should use Python 3.6 (plus await x will fail when you will run the coroutine, since you cannot await on int). |
OK, cool. So, long term, there will be a way to do this (suspend within a generator expression). Thanks for the pointer. |
(As far as awaiting on int, yes, I know how await works, I was focusing on the syntax.) |
Just to add my comment to this 7-years-old never-resolved issue: in PyPy 3.5, which behaves like Python 3.x in this respect, I made the following constructions give a warning. def wrong_listcomp():
return [(yield 42) for i in j]
def wrong_gencomp():
return ((yield 42) for i in j)
def wrong_dictcomp():
return {(yield 42):2 for i in j}
def wrong_setcomp():
return {(yield 42) for i in j} SyntaxWarning: 'yield' inside a list or generator comprehension behaves unexpectedly (http://bugs.python.org/issue10544) The motivation is that none of the constructions above gives the "expected" result. In more details:
For completeness, I think there is no problem with "await" instead of "yield" in Python 3.6. How about fixing CPython to raise SyntaxWarning or even SyntaxError? |
I think it is better to just fix the issue, i.e. make comprehensions be equivalent to for-loops even if they contain The example of |
Let's see if the discussion goes anywhere or if this issue remains in limbo for the next 7 years. In the meantime, if I may humbly make a suggestion: whether the final decision is to give SyntaxError or change the semantics, one or a few intermediate versions with a SyntaxWarning might be a good idea. |
"Just fix the issue" is easier said than done for the same reason that comprehensions were implemented the way they are now: lambda expressions still have to work. That is, we need to maintain the invariant that:
give the same results (respectively) as:
Once you work through the implications of "We need the loop variable to visible to lexically nested scopes, but invisible in the containing scope", you're going to end up with something that looks enough like a nested function that the easiest to implement and explain option is to have it *be* a nested function. I'd be fine with a resolution that forbade yield expressions directly inside implicit scopes, though. |
Also see https://bugs.python.org/issue1660500 for the original Python 3.0 change to hide the iteration variable. While the test suite already covers some interesting scoping edge cases as result of that initial patch, I think one we're currently missing is the nested comprehension case: >>> [[x for x in range(1, i)] for i in range(2, 5)]
[[1], [1, 2], [1, 2, 3]] |
Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect? |
Sorry, I didn't mean to insult anybody. I asked an honest question with an intent to clarify if there's some misunderstanding of the topic that I'm partially responsible for in CPython. |
Yury OK, sorry then this is a misunderstanding from my side. |
NP. Again, sorry if I sounded that way to you. |
Guido, I am not sure about the complete removal, this is probably to radical. What I think we are missing more detailed docs that would be clear about the corner cases. (I already mentioned this in https://bugs.python.org/issue32113) |
I think we all need to calm down a bit. How about not posting about this |
OK, makes sense :-) |
Given the direction of the python-dev thread, should we split this question into two issues? Issue 1: a yield expression inside a comprehension changes the type of the expression result (returning a generator-iterator instead of the expected container type) Issue 2: a yield expression inside a generator expression interacts weirdly with the genexp's implicit yield expression I ask, as it seems to me that issue 1 can be addressed by wrapping the affected cases in an implicit 'yield from' expression, which will both fix the return type of the expression and turn the outer function into a generator (if it isn't one already). (I'm going to put together a proof-of-concept for that idea this weekend) By contrast, the interaction between generator expressions and explicit yield expressions seems intrinsically confusing, so I'm not sure we can do any better than declaring it a syntax error to try to combine them. |
I realised that even without modifying the compiler first, I could illustrate the proposed
So if we decided to make yield-in-a-comprehension imply the use of yield from, we'd only need:
|
No to both. See python-dev. |
Here is a sample of the implementation of the Nick's idea. |
Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much difficulty). The PR does highlight an interesting subtlety though: the easiest way to implement this still allows yield expressions in the outermost iterator, since that gets compiled in a different scope from the rest of the comprehension body (it's evaluated eagerly and passed in to the implicit nested function). I'm thinking we probably don't want to expose that detail to end users, and will instead want to include a second check that prohibits yield expressions in the outermost iterator as well. However, I'm not entirely sure how we could implement such a check, short of adding a new "yield expression prohibited" counter in the AST generation and/or symbol analysis pass. |
No. On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:
|
To clarify, the outermost iterator is marked here:
and it is evaluated before the comprehension proper is started. This is I see no benefit in banning On Sat, Nov 25, 2017 at 10:22 PM, Guido van Rossum <report@bugs.python.org>
|
Cool, if you're OK with that behaviour, it actually makes this a lot easier, since it means:
|
Great. It should be a DeprecationWarning, since we're planning to disallow |
Guido, should we write this change up as a PEP, or are you happy to just cover it as a section in the What's New document for 3.7? |
Note that the DeprecationWarning exception is replaced with a SyntaxError to get a more accurate error report. $ cat yield-gen.py
def f():
return ((yield x) for x in range(3))
$ ./python -Wd yield-gen.py
yield-gen.py:2: DeprecationWarning: 'yield' inside generator expression
return ((yield x) for x in range(3))
$ ./python -We yield-gen.py
File "yield-gen.py", line 2
return ((yield x) for x in range(3))
^
SyntaxError: 'yield' inside generator expression Without this replacement the result would be: $ ./python -We yield-gen.py
DeprecationWarning: 'yield' inside generator expression |
After the tiresome debate I am happy to see this just as a "what's new" entry rather than soliciting more debate with a PEP. (However there may be some existing PEP that suggests it should work? That PEP should be amended with a note that this is being deprecated. But I don't know if there is such a PEP.) |
As far as I'm aware, there's nothing that specifically promises these constructs will do anything in Py3 at all - the existing behaviour is just an accident of implementation arising from the way nested scopes and yield expressions interact in general. Tinkering with "await" in comprehensions and generator expressions would be different, since PEP-530 actually does make promises about how we expect that to work. |
OK, great. |
With Serhiy's patch merged, I'm marking this as resolved. Thanks all! https://bugs.python.org/issue32189 is the follow-up issue to turn the warning into an unconditional SyntaxError in 3.8. |
PR 4676 backports warnings to 2.7 in Py3k mode. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: