can't run close through itertools.chain on inner generator

[dangyogi]

BPO	3842
Nosy	@rhettinger, @amauryfa

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2008-09-12.21:31:01.293>
created_at = <Date 2008-09-11.19:03:14.643>
labels = ['extension-modules', 'type-feature']
title = "can't run close through itertools.chain on inner generator"
updated_at = <Date 2008-09-12.21:31:01.258>
user = 'https://bugs.python.org/dangyogi'

bugs.python.org fields:

activity = <Date 2008-09-12.21:31:01.258>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2008-09-12.21:31:01.293>
closer = 'rhettinger'
components = ['Extension Modules']
creation = <Date 2008-09-11.19:03:14.643>
creator = 'dangyogi'
dependencies = []
files = []
hgrepos = []
issue_num = 3842
keywords = []
message_count = 4.0
messages = ['73052', '73079', '73118', '73141']
nosy_count = 3.0
nosy_names = ['rhettinger', 'amaury.forgeotdarc', 'dangyogi']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3842'
versions = ['Python 3.1', 'Python 2.7']

[dangyogi]

There is no way to get generators to clean up (run their 'finally'
clause) when used as an inner iterable to chain:

>>> def gen(n):
...     try:
...         # do stuff yielding values
...     finally:
...         # clean up
>>> c = chain.from_iterable(map(gen, (1,2,3)))
>>> next(c)
0
>>> # done with c, but can't clean up inner gen!

Could you add a 'close' method to itertools.chain that would call close
(if present) on both the inner and other iterable? Then clean up could
be done as:

>>> with closing(chain.from_iterable(map(gen, (1,2,3)))) as c:
...    next(c)
>>> # generator finalized by "with closing"!

[amauryfa]

You somehow must tell the iterator that you are done with it.
This is best done by a "del c" in your first snippet, since c is the
last reference to the iterator.
To do this in a function (or a context manager), the trick is to wrap
the map() iterator inside a generator function.

def wrapiterator(it):
    for x in it:
        yield x

Then your sample becomes:
>>> with closing(wrapiterator(chain.from_iterable(
...                  map(gen, (1,2,3))))) as c:
...    next(c)

Which of course can be rewritten as:

def closingiterator(it):
    def wrapper():
        for x in it:
            yield x
    return closing(wrapper())

>>> with closingiterator(chain.from_iterable(map(gen, (1,2,3))))) as c:
...    next(c)

This works because the only reference to the map() iterator is held by
the wrapper generator function. Calling its close() method will
terminate the function, delete its locals, ultimately call the
deallocator of the gen() iterator, which will fire the "finally" block.

Your proposal implies that all c-based iterators need to grow a "close"
method. This is not practical, and is best simulated with this wrapper
generator: close()ing the wrapper will (recursively) destroy the iterators.
If this also works for you, I suggest to close this issue.

[dangyogi]

What you propose is not portable code. It won't work on jython or
ironpython or pypy because they don't have reference counting garbage
collectors that immediately reclaim objects.

Also I'm not asking that all c-based iterators grow a 'close' method
(though this would be nice! :-) ).

For all other c-based iterators, you can do:

with closing(gen()) as argument:
    for x in map(fn, argument): ...

which is portable.

So I am only asking that 'close' be added to itertools.chain because
there is no portable solution there.

-bruce

Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> You somehow must tell the iterator that you are done with it.
> This is best done by a "del c" in your first snippet, since c is the
> last reference to the iterator. 
> To do this in a function (or a context manager), the trick is to wrap
> the map() iterator inside a generator function.
>
> def wrapiterator(it):
>     for x in it:
>         yield x
>
> Then your sample becomes:
>   
>>>> with closing(wrapiterator(chain.from_iterable(
>>>>         
> ...                  map(gen, (1,2,3))))) as c:
> ...    next(c)
>
>
> Which of course can be rewritten as:
>
> def closingiterator(it):
>     def wrapper():
>         for x in it:
>             yield x
>     return closing(wrapper())
>
>   
>>>> with closingiterator(chain.from_iterable(map(gen, (1,2,3))))) as c:
>>>>         
> ...    next(c)
>
>
> This works because the only reference to the map() iterator is held by
> the wrapper generator function. Calling its close() method will
> terminate the function, delete its locals, ultimately call the
> deallocator of the gen() iterator, which will fire the "finally" block.
>
> Your proposal implies that all c-based iterators need to grow a "close"
> method. This is not practical, and is best simulated with this wrapper
> generator: close()ing the wrapper will (recursively) destroy the iterators.
> If this also works for you, I suggest to close this issue.
>

[rhettinger]

Sorry, am going to reject this. The use cases are somewhat uncommon
and I don't want to clutter-up tools that need to remain as simple as
possible. The pure python code for chain() is so simple that it's not
hard to roll your own version as needed.