Add contextlib.ExitStack

[nikratio]

BPO	13585
Nosy	@rhettinger, @ncoghlan, @pitrou, @giampaolo, @merwok, @meadori, @Julian, @ericsnowcurrently, @smarnach, @jdemeyer
Files	CleanupManager_patch_v1.diff CleanupManager_patch_v2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2012-05-21.12:54:58.701>
created_at = <Date 2011-12-12.01:49:50.244>
labels = ['type-feature', 'library']
title = 'Add contextlib.ExitStack'
updated_at = <Date 2018-04-12.05:25:34.535>
user = 'https://bugs.python.org/nikratio'

bugs.python.org fields:

activity = <Date 2018-04-12.05:25:34.535>
actor = 'jdemeyer'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2012-05-21.12:54:58.701>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2011-12-12.01:49:50.244>
creator = 'nikratio'
dependencies = []
files = ['23923', '23933']
hgrepos = []
issue_num = 13585
keywords = ['patch']
message_count = 34.0
messages = ['149268', '149269', '149271', '149272', '149273', '149303', '149352', '149353', '149354', '149366', '149372', '149373', '149376', '149377', '149378', '149380', '149384', '149393', '150011', '150012', '150017', '150028', '150057', '150063', '150065', '150071', '150080', '150090', '150092', '157386', '159754', '161271', '315201', '315213']
nosy_count = 12.0
nosy_names = ['rhettinger', 'ncoghlan', 'pitrou', 'giampaolo.rodola', 'eric.araujo', 'nikratio', 'meador.inge', 'Julian', 'python-dev', 'eric.snow', 'smarnach', 'jdemeyer']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13585'
versions = ['Python 3.3', 'Python 3.4']

[nikratio]

I'd like to propose addding the CleanupManager class described in http://article.gmane.org/gmane.comp.python.ideas/12447 to the contextlib module. The idea is to add a general-purpose context manager to manage (python or non-python) resources that don't come with their own context manager.

Example code:

with CleanupManager() als mngr:
    tmpdir = tempfile.mkdtemp()
    mngr.register(shutil.rmtree(tmpdir))
    # do stuff with tmpdir

# shutil.rmtree() get's called automatically when the block is over

Note that mkdtemp() could of course also be changed to become its own context manager. The idea is to provide a general facility for this kind of problem, so it doesn't have to be reinvented whenever a module provides a ressource without its own context manager. Other possible uses are of course ressources that are completely external to Python,
e.g. anything allocated with a subprocess (think of subprocess.check_call('mount'))/

I'll be happy to make a proper patch with documentation and testcases from Jan's code. As a matter of fact, I'll probably start working out it right now, so please let me know quickly if this doesn't have a chance of getting accepted.

[nikratio]

Here's the first part of the patch with the implementation. I'll add tests and documentation as soon as someone confirms that the idea & API is okay.

[rhettinger]

I would like to see this posted as a recipe before being put in the standard library. It needs a chance to mature and to demonstrate that people will want to use it.

FWIW, the example is problematic in a couple of ways. The registration of shutil.rmtree(tmpdir) will run *before* mngr register is called.

Also, it doesn't take advantage of any of the with-statement features. It doesn't show any advantage over a standard try/finally which is arguably cleaner:

    tmpdir = tempfile.mkdtemp()
    try:
        # do stuff with tmpdir
    finally:
        shutil.rmtree()

Also, I suspect that the CleanupManager would be an error-prone construct because the registration occurs somewhere after the with-statement is set-up, possibly resulting in errors if there is an early, pre-registration failure.

[nikratio]

Not sure what you mean with "posted as a recipe" -- are you thinking of a specific website/mailing list?

Example: which one do you mean? The one in the issue or the one in the patch?

With statement: what advantages do you have in mind?

Try/finally: I think the patch and the discussion in python-ideas talk about the advantage over try/finally. IMO the two most important points are: (1) avoids deep and pointless indendation for multiple ressources, (2) keeps logically connected lines (allocation+cleanup) closely together in the source instead of splitting them far apart like try/finally.

error-prone: not sure if I understand you correctly. If there is an error prior to registration, the callback will not be called (that's a feature). To what kind of errors could that lead?

Sorry for basically asking you to re-explain every sentence, but I honestly don't understand most of your message.

[rhettinger]

'''
Example code:

with CleanupManager() als mngr:
    tmpdir = tempfile.mkdtemp()
    mngr.register(shutil.rmtree(tmpdir)) <-- this makes the call right away
    # do stuff with tmpdir
'''

The part of my note that should be clear is that the idea and code need to prove itself before being added to the standard library. So far, there has been zero demand for this and I've not seen code like it being used in the wild. AFAICT, it is not demonstrably better than a straight-forward try/finally.

[nikratio]

On 12/11/2011 10:17 PM, Raymond Hettinger wrote:

Raymond Hettinger <raymond.hettinger@gmail.com> added the comment:

'''
Example code:

with CleanupManager() als mngr:
tmpdir = tempfile.mkdtemp()
mngr.register(shutil.rmtree(tmpdir)) <-- this makes the call right away
# do stuff with tmpdir
'''

Oh, of course. That is fixed in the patch. I couldn't find a way to edit
the message in the tracker.

The part of my note that should be clear is that the idea and code need to prove itself before being added to the standard library. So far, there has been zero demand for this and I've not seen code like it being used in the wild. AFAICT, it is not demonstrably better than a straight-forward try/finally.

I think it has the same advantages over try...finally as any use of
'with' has. CleanupManager would allow to use 'with' instead of
'try..finally' in many cases where this is currently not possible, so
unless the introduction of 'with' itself was a mistake, I think this is
just taking a step along the path that has already been chosen.

Best,

-Nikolaus

[smarnach]

There is actually a second thread on python-ideas on a very similar topic, see

http://mail.python.org/pipermail/python-ideas/2011-December/013021.html

The two main advantages of the proposed CleanupManager over try/finally blocks are

1. You can add a clean-up function conditionally. In a try/finally block, you would need a flag, which would scatter the single idea more across the code. Example:

    with CleanupManager() als mngr:
        if f is None:
            f = open("some_file")
            mngr.register(f.close)
        # do something with f (possibly many lines of code)

seems much clearer to me than

    f_needs_closing = False
    if f is None:
        f = open("some_file")
        f_needs_closing = True
    try:
        # do something with f (possibly many lines of code)
    finally:
        if f_needs_closing:
            f.close()

The first version is also much more in the spirit of context managers. You state at the beginning "we open the file, and guarantee that it will be closed", and we know right from the start that we don't have to bother with it again.

2. CleanupManager could replace several nested try/finally blocks, which again might lead to more readable code.

On the other hand, people who never saw ContextManager before will have to look it up, which will impair readability for those people.

Adding this as a cookbook recipe first seems like a good idea.

[nikratio]

On 12/12/2011 03:31 PM, Sven Marnach wrote:

Adding this as a cookbook recipe first seems like a good idea.

This is the second time that this is mentioned, so I would be very
grateful if someone could elucidate what "adding as a cookbook recipe"
means :-). Is there an official Python cookbook somewhere?

Best,

-Nikolaus

[ericsnowcurrently]

Check out: http://code.activestate.com/recipes/

[Julian]

For reference, the implementation that I posted in the other thread is:

    @contextlib.contextmanager
    def maybe(got, contextfactory, *args, checkif=bool, **kwargs):
        if checkif(got):
            yield got
        else:
            with contextfactory(*args, **kwargs) as got:
                yield got

which produces code like:

def fn(input_file=None):
    with maybe(input_file, open, "/default/input/file/location/"):
        for line in input_file:
            print line

For the example given here in the OP, since rmtree isn't a contextmanager it'd require wrapping it in one first.

I could probably understand if there was desire for these to be a recipe instead. The advantage (if any) of the above over this class is that it keeps context managers sorta looking like context managers. Here if you wanted to write my example it'd require writing code like

with ContextManager() as mgr:
    foo = ctxtmgr()
    mgr.register(foo.__close__)

which doesn't look too great :/

The class though does have wider scope though, since it doesn't necessarily only need a context manager, it can do cleanup for anything, which I guess is nice.

[ncoghlan]

Given the existence of tempfile.TemporaryDirectory in recent Python versions, I suggest finding a new cleanup function example that doesn't duplicate native stdlib functionality :)

I do see value in the feature itself though - I believe the precedent of both the atexit module [1] and unittest.TestCase.addCleanup() [2] shows how it can be useful to have a standard way to accumulate a sequence of "undo" operations for invocation at a later time.

In particular, it's a much better fit than nested with statements are for *optional* resources - you can make the with statement unconditional (setting up the cleanup manager), optionally add the cleanup methods, and avoid needing to have two copies of your actual invocation code (one inside a with statement and one without) or having a delayed check in a finally block.

It codifies the fairly common "if this resource was acquired, make sure it is released" idiom in a way that with statements and try/finally just don't handle neatly. By using an incremental API, it also avoids the traps associated with the ultimately misguided "contextlib.nested()" design.

However, I suggest using an API that strictly follows the "register only" model employed by both of the existing mechanisms. In addition, any solution provided as part of contextlib should interoperate nicely with existing context managers - it should be trivial to rewrite a nested with statement to be based on CleanupManager instead.

Accordingly, I would give the manager the following public methods:

register_exit() (only accepts callbacks with the __exit__ signature)
register() (equivalent to TestCase.addCleanup)
enter_context() (accepts actual context managers)
close() (equivalent to TestCase.doCleanups)

register_exit() would be the base callback registration method. It would accept only callbacks with the same signature as __exit__ methods.

    def register_exit(self, exit):
        self._callbacks.append(exit)
        return exit # Allow use as a decorator

register() would wrap arbitrary callbacks to support the __exit__ method signature:

    def register(self, _cb, *args, **kwds):
        def _wrapper(exc_type, exc, tb):
            return _cb(*args, **kwds)
        return self.register_exit(_wrapper)

enter_context() would work as follows:

    def enter_context(self, cm):
        result = cm.__enter__()
        self.register_exit(cm.__exit__)
        return result

close() would look like:

    def close(self):
        self.__exit__(None, None, None)

And finally, __exit__() itself would be:

    def __exit__(self, *exc_details):
        def _invoke_next_callback(exc_details):
            # Callbacks are removed from the list in FIFO order
            # but are actually *invoked* in LIFO order
            cb = self._callbacks.pop(0)
            if not self._callbacks:
                # Innermost callback is invoked directly
                return cb(exc_type, exc, tb)
            # Use try-finally to ensure this callback still gets
            # invoked even if an inner one fails
            try:
                inner_result = _invoke_next_callback()
            except:
                cb_result = cb(*sys.exc_info())
                # Check if this cb suppressed the inner exception
                if not cb_result:
                    raise
            else:
                # Check if inner cb suppressed the original exception
                if inner_result:
                    exc_details = (None, None, None)
                cb_result = cb(*exc_details)
            return cb_result
        _invoke_next_callback(exc_details)

An example using a cleanup manager to handle multiple files, one of which is optional:

    with contextlib.CleanupManager() as cm:
        source = cm.enter_context(open(source_fname))
        if dest_fname is not None:
            dest = cm.enter_context(open(dest_fname))
            _write_to_dest = dest.write
        else:
            def _write_to_dest(line): pass
        for line in source:
            _write_to_dest(line)
            yield line

[1] http://docs.python.org/library/atexit
[2] http://docs.python.org/library/unittest#unittest.TestCase.addCleanup

[rhettinger]

I think you guys need to post your code somewhere (perhaps on PyPi or the ASPN Cookbook). It needs to mature beyond the stage of "I just whipped-up this code and think it would be great if everybody used it".

I've seen nothing like this being used in production code (code published on the net or at one of my clients). Context managers have been around for a while, so if this were a real need, we would expect to see people already using something like this (for example, namedtuples got introduced to the standard library upon seeing many, many reinventions of the concept and we were able to consolidate the best features from each).

Design of the a feature in the standard library should be driven by examples of real world code that would be improved with the new feature. The design should also be informed by the experience of teaching people how to use it and seeing what they do (lots of technically correct ideas get shot down because it turns out that incorrect usage is common (a bug factory like the % formatting operator) or that people have a hard time learning and remembering the feature.

The core problem is that it is easier to add things to the standard library than to take them out if they prove to be a bad idea. Accordingly, we need to be *really sure* that this is a good idea, that it will *improve* real world code, that people learn, understand, and remember it easily, and that is doesn't impair readability.

The example code from Nikolaus has a bug in it -- that is worrisome and may suggest that we are better-off without this being in the standard library.

[ncoghlan]

TestCase.setUp() and TestCase.tearDown() were amongst the precursors to__enter__() and __exit__(). addCleanUp() fills exactly the same role here - and I've seen *plenty* of positive feedback directed towards Michael for that addition to the unittest API.

For individual one-off cases, a flag variable and an if statement inside a finally block is an adequate, but not ideal, solution, because it suffers from all the readability and auditability problems of *any* try/finally based solution. It's particularly annoying when an object *does* support the context management protocol, but I can't use a with statement simply because I don't *always* need (and/or own) that resource (this kind of thing happens in a few places in runpy, since the behaviour changes depending on whether or not runpy created temporary objects for itself or was given objects as arguments)

Custom context managers are typically a bad idea in these circumstances, because they make readability *worse* (relying on people to understand what the context manager does). A standard library based solution, on the other hand, offers the best of both worlds:

• code becomes easier to write correctly and to audit for correctness (for all the reasons with statements were added in the first place)
• the idiom will eventually become familiar to all Python users

If other "cleanup function" registration APIs didn't already exist, I'd agree with you that this needed further exposure. However, I simply don't agree that's the case - atexit and addCleanup provide your field testing, the rest of the design is just a matter of integrating those concepts with the context management protocol.

Indeed, one of the objections I received after we deprecated contextlib.nested() was that you couldn't easily pass a programmatically generated list of resources to nested with statements. Given contextlib.CleanupManager it becomes trivial:

    with contextlib.CleanupManager as cm:
        files = [cm.enter_context(open(fname)) for fname in names]
        # All files will be closed when we leave the context

I can take this up on python-dev if you want, but I hope to persuade you that the desire *is* there, it's just that the workarounds for the lack of this functionality involve avoiding the context management protocol entirely:

try:
    files = [open(fname) for fname in names]
    # Are all files closed when we're done?
    # I dunno, scroll down past the algorithm code to check!


    # Avoiding this would be good for all the reasons the
    # with statement was added in the first place

finally:
    for f in files:
        f.close()

[ncoghlan]

Given the history of API design errors in contextlib (cf. contextlib.nested in general, making contextlib._GeneratorContextManager a subclass of contextlib.ContextDecorator), I've realised Raymond is right in wanting to see this idea more thoroughly vetted before it gets added to the standard library.

Accordingly, I plan to create a 'unittest2' style backport library for contextlib (imaginatively named 'contextlib2') and prototype the concept further there.

[ncoghlan]

In the meantime, I put my version up as a cookbook recipe: http://code.activestate.com/recipes/577981-cleanupmanager-for-with-statements/

[rhettinger]

Thanks Nick. You're awesome.

[ncoghlan]

And the backport: http://contextlib2.readthedocs.org/

I haven't tested on anything other than 2.7 as yet - I have an account request in train with the Shining Panda folks, so I'll set up multi-version CI for this project (along with a couple of others) once that goes through.

[smarnach]

I think that the fact that Nick got the code to close multiple files wrong underlines that it is difficult to get right currently. Nick's code

try:
    files = [open(fname) for fname in names]
    # ...
finally:
    for f in files:
        f.close()

only closes the files if all of them were opened successfully. Moreover, file.close() can fail for various reasons, which would result in all remaining files being left open. When we fix both problems, the code becomes

try:
    files = []
    for fname in names:
        files.append(open(fname))
    # ...
finally:
    for f in files:
        try:
            f.close()
        except IOError:
            pass

I think everyone will agree that the version using 'CleanupManager' is nicer. To be fair, we should note that the need to open many files simultaneously is not very common -- usually, we can make to with opening the files one by one.

[merwok]

The idea is to add a general-purpose context manager to manage (python
or non-python) resources that don't come with their own context manager.

I read the thread back on python-ideas and didn’t like the idea, without knowing exactly why—maybe a feeling that it’s too generic and less clean than adding context management support to the classes directly. My opinion is only that of a user, I’m less qualified than Nick or Raymond for this module.

In the passage I quoted, I don’t understand what is meant by “non-Python resources”.

[nikratio]

On 12/21/2011 11:46 AM, Éric Araujo wrote:

Éric Araujo <merwok@netwok.org> added the comment:

> The idea is to add a general-purpose context manager to manage (python
> or non-python) resources that don't come with their own context manager.

In the passage I quoted, I don’t understand what is meant by “non-Python resources”.

Think about e.g. mounting a file system.

Best,

-Nikolaus

[merwok]

> In the passage I quoted, I don’t understand what is meant by “non-Python resources”.
Think about e.g. mounting a file system.

Ah, ok. In that case there would still be a Python-level object (just like a Python file object will also release the OS-level handle when it’s closed).

[nikratio]

On 12/21/2011 12:03 PM, Éric Araujo wrote:
> 
> Éric Araujo <merwok@netwok.org> added the comment:
> 
>>> In the passage I quoted, I don’t understand what is meant by “non-Python resources”.
>> Think about e.g. mounting a file system.
> 
> Ah, ok.  In that case there would still be a Python-level object (just like a Python file object will also release the OS-level handle when it’s closed).

I don't think so.

subprocess.check_call(['mount', 'bla', '/mnt'])

Allocates the resource (mounts the file system) but doesn't leave you
with any Python object.

Best,

-Nikolaus

[ncoghlan]

My earlier descriptions here aren't really adequate - as soon as I started putting contextlib2 together, this CleanupManager idea quickly morphed into ContextStack [1], which is a far more powerful tool for manipulating context managers in a way that doesn't necessarily correspond with lexical scoping in the source code. Using the "collection of files" example:

    with ContextStack() as stack:
        files = [stack.enter_context(open(fname)) for fname in filenames]
        # All opened files will automatically be closed at the end of
        # the with statement, even if attempts to open files later
        # in the list throw an exception

Or the optional resource use case:

    with ContextStack() as stack:
        if resource is None:
            resource = stack.enter_context(make_default_resource())
        # If we created it, the resource will be cleaned up
        # Otherwise, it will be left alone

The "cleanup only if the operation fails" use case (coming in v0.3 [2])

    def open_files(*filenames):        
        """Returns an (opened_files, context_stack) 2-tuple

    The context stack will automatically close all of the opened files.
    If any file fails to open, all previously opened file handles will be released immediately.
    """
    with ContextStack() as stack:
        files = [stack.enter_context(open(fname)) for fname in filenames]
        return files, stack.preserve()

The "don't repeat your __exit__ code in __enter__" use case:

    def __enter__(self):
        resource = self._acquire_resource()
        with ContextStack() as stack:
            stack.register_exit(self.__exit__)
            self._check_resource_validity()
            stack.preserve() # All good!
        return resource

(In 0.3, register_exit() will probably check for __exit__ attributes automatically, so it will accept both callables and context managers)

[1] http://contextlib2.readthedocs.org/en/latest/index.html#contextlib2.ContextStack
[2] https://bitbucket.org/ncoghlan/contextlib2/issue/1/add-a-preserve-method-to-relinquish

[ncoghlan]

Updated issue title to reflect what I'll eventually be proposing (once the ContextStack API has had a chance to mature on PyPI as part of contextlib2)

[rhettinger]

"I put my version up as a cookbook recipe: http://code.activestate.com/recipes/577981-cleanupmanager-for-with-statements/"

One other idea is to model what I've done with the itertools docs by adding a recipe section. I used it as an incubator for possible new itertools; as a set of tested cut-and-pasteable recipes; and to serve as an instructive guide for learning about how to build iterators. A nice side benefit of posting code in the docs is that I was free to change, improve, or remove the recipes over time (the contrasts with real additions to the standard library which are hard to change once they are released).

[ncoghlan]

Yeah, adding a "Recipes" section for contextlib2 is definitely on my to-do list (along with adding more examples in general).

[pitrou]

with ContextStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]

I find this a bit distasteful. Cleaning up resources now needs something called a "ContextStack" and an "enter_context" method call. There's a bit too much terminology, and it looks like a poor man's equivalent of Go's "defer" statement:
http://blog.golang.org/2010/08/defer-panic-and-recover.html

I like unittest's addCleanup mechanism better.

[ncoghlan]

ContextStack is intended to be a building block that makes it easy to compose context managers and other callbacks, not necessarily an API you'd regularly use directly. For example, given ContextStack (as currently implemented in contextlib2), it's trivial to write your own higher level cleanup API:

    @contextmanager
    def cleanup(cb=None, *args, **kwds):
        with ContextStack() as stack:
            if cb is not None:
                stack.register(cb, *args, **kwds):
            yield stack

If you only have one callback, you could supply it directly as an argument to cleanup(), otherwise you could make multiple register() calls on the returned stack object.

The idea is to implement the necessary __exit__() logic that makes it feasible to compose context managers and other callbacks just once, then let people explore the possibilities in terms of the higher level APIs that it makes easy.

[ncoghlan]

The comparison to Go's defer statement is interesting, though: ContextStack.register basically *is* the same as defer. While a nested with statement is a better option in Python, if we ignore that for the moment, you *could* write that simply copying example:

    with ContextStack() as stack:
        src = open(source)
        stack.register(src.close)
        dest = open(destination, 'w')
        stack.register(dest.close)
        copy(src, dest)

However, since ContextStack is just an ordinary context manager class, you have a lot of flexibility in what you do with it (particularly once I add the preserve() API to let you deliberately *skip* the cleanup steps by transferring them to a new ContextStack object)

[ncoghlan]

I'm unlikely to add the contextlib2.ContextStack API as written. I aim to have an updated variant (called contextlib.CallbackStack) available for alpha 3 in May:
https://bitbucket.org/ncoghlan/contextlib2/issue/8/rename-contextstack-to-callbackstack-and

[ncoghlan]

Latest draft of API is here: http://contextlib2_dev.readthedocs.org/en/latest/index.html#contextlib2.ExitStack

An updated version of the "I forgot I could use multiple context managers in a with statement" example:

    with ExitStack() as stack:
        src = open(source)
        stack.callback(src.close)
        dest = open(destination, 'w')
        stack.callback(dest.close)
        copy(src, dest)

The example of opening a collection of files remains unchanged (aside from s/ContextStack/ExitStack/).

Also see: http://contextlib2_dev.readthedocs.org/en/latest/index.html#replacing-any-use-of-try-finally-and-flag-variables

[python-dev]

New changeset 8ef66c73b1e1 by Nick Coghlan in branch 'default':
Close bpo-13585: add contextlib.ExitStack to replace the ill-fated contextlib.nested API
http://hg.python.org/cpython/rev/8ef66c73b1e1

[jdemeyer]

Why this?

_exit_wrapper.__self__ = cm

It seems that you are trying to create something which is exactly like a method except that it's not a method. Is there any reason to not use an actual method? It would actually simplify the code.

I ask because assigning __self__ might break after PEP-575.

[jdemeyer]

Follow-up: https://bugs.python.org/issue33265