Nested generator terminates prematurely

[ygale]

BPO	955772
Nosy	@mwhudson, @arigo, @rhettinger, @terryjreedy

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2004-06-01.17:08:53.000>
created_at = <Date 2004-05-18.10:02:49.000>
labels = ['interpreter-core', 'invalid']
title = 'Nested generator terminates prematurely'
updated_at = <Date 2004-06-01.17:08:53.000>
user = 'https://bugs.python.org/ygale'

bugs.python.org fields:

activity = <Date 2004-06-01.17:08:53.000>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2004-05-18.10:02:49.000>
creator = 'ygale'
dependencies = []
files = []
hgrepos = []
issue_num = 955772
keywords = []
message_count = 10.0
messages = ['20823', '20824', '20825', '20826', '20827', '20828', '20829', '20830', '20831', '20832']
nosy_count = 5.0
nosy_names = ['mwh', 'arigo', 'rhettinger', 'terry.reedy', 'ygale']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue955772'
versions = []

[ygale]

def g(x, y):
  for i in x:
    for j in y:
      yield i, j

r2 = (0, 1)
[e for e in g(r2, g(r2, r2))]

Expected result:

[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)),
(1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))]

Actual result:

[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))]

[ygale]

Logged In: YES
user_id=1033539

Trying again to get the indentation correct:

def g(x, y):
  for i in x:
    for j in y:
      yield i, j

[mwhudson]

Logged In: YES
user_id=6656

Um. I think the answer to this is "generators are not
reiterable".

[ygale]

Logged In: YES
user_id=1033539

Too bad. What exactly is the restriction?
I didn't find anything in the docs. And things
like this often do work and are useful.

For example:

def primes():
  yield 2
  for n in count(3):
    for p in primes():
      if p > sqrt(n):
        yield n
        break
      if n % p == 0:
        break

[mwhudson]

Logged In: YES
user_id=6656

Well, it's impossible in general. You'd have to store any
arguments the generator took somewhere too, wouldn't you?

What about things like:

def foo(aList):
      while aList:
           yield aList.pop()

?

[ygale]

Logged In: YES
user_id=1033539

Python functions can be called recursively
in general. They know how to save their
local namespace in a separate
frame for each call. That includes arguments,
since the arguments live in the local namespace
of the function.

Generator functions also seem to be supported,
as my example shows.

There is a restriction on a generator object
that you may not call its next() method again
while a previous call to next() is still running.

But this is definitely not a case of that
restriction - we have two separate generator
instances, and each ought to have its own frame.

If there is some other restriction, I think it ought
to be documented. And if possible, it should
raise an exception, like the other restriction.

This smells like a bug to me, though.

[arigo]

Logged In: YES
user_id=4771

Your issue is that you only create a total of two generator
instances. The 'inner' one is immediately exhausted.
Afterwards, this same instance is used again on other 'for'
loops but this has no effect, as it has already been exhausted.

The difference is the same as between r2 and iter(r2). If
you do that:

it = iter(r2)
for x in it: print x
for x in it: print x

the second loop will be empty beause the first loop has
exhausted the iterator. Generators are iterators (like it)
and not sequences (like r2).

Using the same iterator on several 'for' loops is useful,
though (e.g. if the first loop can be interrupted with
'break'), so there is no way your code could raise an
exception, short of saying that it is not allowed to call
next() on already-exhausted iterators -- this would be too
big a change.

[rhettinger]

Logged In: YES
user_id=80475

Marking this as invalid and closing.

Sorry, non-re-iterability is documented fact of life in the
world of generators and iterators.

The work arounds include making the inner generator into a
list or re-instantiating a new generator on every loop:

def g(x, y):
for i in x
for j in g(x, y)
yield i, j

[ygale]

Logged In: YES
user_id=1033539

OK. I can get the semantics I want using the following:

def g(x, y):
  for i in x:
    for j in y:
      yield i, j
g = restartable(g)

where I have defined:

class restartable:
  def __init__(self, genfn):
    self.genfn = genfn
  def __call__(self, *args):
    return restartable_generator(self.genfn, *args)

class restartable_generator:
  def __init__(self, genfn, *args):
    self.genfn = genfn
    self.args = args
  def __iter__(self):
    return self.genfn(*self.args)

[terryjreedy]

Logged In: YES
user_id=593130

Unless you are generating a very long list, you worked too
hard. But not as cute.

>>> def g(x, y):
...   y = list(y)
...   for i in x:
...     for j in y:
...       yield i, j
...
>>> r2 = (0, 1)
>>> [e for e in g(r2, g(r2, r2))]
[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)),
 (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))]