Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a "no-op" (null) context manager to contextlib #54258

Closed
hniksic mannequin opened this issue Oct 8, 2010 · 43 comments
Closed

Add a "no-op" (null) context manager to contextlib #54258

hniksic mannequin opened this issue Oct 8, 2010 · 43 comments
Assignees
Labels
3.7 3.8 stdlib type-feature

Comments

@hniksic
Copy link
Mannequin

@hniksic hniksic mannequin commented Oct 8, 2010

BPO 10049
Nosy @birkenfeld, @rhettinger, @ncoghlan, @pitrou, @vstinner, @giampaolo, @hniksic, @merwok, @bitdancer, @voidspace, @durban, @albertz, @Jesse-Bakker
PRs
  • #4464
  • Files
  • fl
  • nullcontext.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ncoghlan'
    closed_at = <Date 2017-11-23.00:27:28.912>
    created_at = <Date 2010-10-08.11:13:44.343>
    labels = ['3.7', '3.8', 'type-feature', 'library']
    title = 'Add a "no-op" (null) context manager to contextlib'
    updated_at = <Date 2017-11-23.00:27:28.910>
    user = 'https://github.com/hniksic'

    bugs.python.org fields:

    activity = <Date 2017-11-23.00:27:28.910>
    actor = 'ncoghlan'
    assignee = 'ncoghlan'
    closed = True
    closed_date = <Date 2017-11-23.00:27:28.912>
    closer = 'ncoghlan'
    components = ['Library (Lib)']
    creation = <Date 2010-10-08.11:13:44.343>
    creator = 'hniksic'
    dependencies = []
    files = ['19158', '19176']
    hgrepos = []
    issue_num = 10049
    keywords = ['patch']
    message_count = 43.0
    messages = ['118181', '118183', '118184', '118185', '118186', '118187', '118189', '118190', '118193', '118194', '118195', '118196', '118200', '118274', '118406', '118409', '118448', '118450', '118453', '118599', '118616', '118626', '119514', '152909', '153046', '153049', '175656', '175662', '281177', '281180', '281195', '281200', '281215', '281234', '281264', '281556', '305951', '305952', '306001', '306002', '306266', '306770', '306771']
    nosy_count = 18.0
    nosy_names = ['georg.brandl', 'rhettinger', 'blais', 'ncoghlan', 'pitrou', 'vstinner', 'giampaolo.rodola', 'hniksic', 'eric.araujo', 'r.david.murray', 'michael.foord', 'daniel.urban', 'Albert.Zeyer', 'piotr.dobrogost', 'Alexander.Jones', 'DLitz', 'Martin Blais', 'jbakker']
    pr_nums = ['4464']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue10049'
    versions = ['Python 3.7', 'Python 3.8']

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Oct 8, 2010

    I find that I frequently need the "null" (no-op) context manager. For example, in code such as:

    with transaction or contextlib.null():
        ...

    Since there is no easy expression to create a null context manager, we must resort to workarounds, such as:

    if transaction:
        with transaction:
            ... code ...
    else:
        ... duplicated code ...

    (with the usual options of moving the duplicated code to a function—but still.)

    Or by creating ad-hoc null context managers with the help of contextlib.contextmanager:

    if transaction is None:
        transaction = contextlib.contextmanager(lambda: iter([None])()
    with transaction:
        ...

    Adding a "null" context manager would be both practical and elegant. I have attached a patch for contextlib.py

    @hniksic hniksic mannequin added stdlib type-feature labels Oct 8, 2010
    @vstinner vstinner changed the title Add the null context manager to contextlib Add a "no-op" (null) context manager to contextlib Oct 8, 2010
    @voidspace
    Copy link
    Contributor

    @voidspace voidspace commented Oct 8, 2010

    +1

    Looks like a reasonable use case.

    @voidspace
    Copy link
    Contributor

    @voidspace voidspace commented Oct 8, 2010

    Patch is missing tests and documentation.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Oct 8, 2010

    About your patch:

    • __enter__() might return self instead of None... i don't really know which choice is better. "with Null() as x:" works in both cases
    • __exit__() has no result value, "pass" is enough
    • I don't like "Null" name, I prefer "Noop" (NoOperation, NoOp, ...) or something else

    @merwok
    Copy link
    Member

    @merwok merwok commented Oct 8, 2010

    I also find the Null/_null/null affair confusing.

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Oct 8, 2010

    @contextlib.contextmanager
    def null():
        yield

    Do we really need to add this to the stdlib?
    Previous proposals to add an "identity function" or "no-op function" have always be refused. This one seems even less useful.

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Oct 8, 2010

    Thank you for your comments.

    @michael: I will of course write tests and documentation if there is indication that the feature will be accepted for stdlib.

    @antoine: it is true that a null context manager can be easily defined, but it does requires a separate generator definition, often repeated in different modules. This is markedly less elegant than just using contextlib.null() in an expression.

    I'm not acquainted with the history of identity function requests, but note that the identity function can be defined as an expression, using simply lambda x: x. The equivalent expression that evaluates to a null context manager is markedly more convoluted, as shown in my report.

    @Éric: The Null/_null/null distinction is an optimization that avoids creating new objects for something that is effectively a singleton. It would be perfectly reasonable to define contextlib.null as Antoine did, but, this being stdlib, I wanted the implementation to be as efficient as (reasonably) possible.

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Oct 8, 2010

    @antoine: it is true that a null context manager can be easily
    defined, but it does requires a separate generator definition, often
    repeated in different modules. This is markedly less elegant than
    just using contextlib.null() in an expression.

    But you can use mymodule.null() where mymodule is a module gathering
    common constructs of yours.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Oct 8, 2010

    The difference here is the one pointed out in the original post: for a function, you usually only care about having a value, so if you don't want to call it, you can just swap in a None value instead. If you need an actual callable, then "lambda:None" fits the bill.

    The with statement isn't quite so forgiving. You need a genuine context manager in order to preserve the correct structure in the calling code. It isn't intuitively obvious how to do that easily. While not every 3-line function needs to be in the standard library, sometimes they're worth including to aid discoverability as much as anything else.

    However, I don't see the point in making it a singleton and the name should include the word "context" so it doesn't becoming ambiguous when referenced without the module name (there's a reason we went with contextlib.contextmanager over contextlib.manager).

    Something like:

    class nullcontext():
        """No-op context manager, executes block without doing any additional processing.
    Used as a standin if a particular block of code is only sometimes
    used with a normal context manager:
    
          with optional_cm or nullcontext():
              # Perform operation, using the specified CM if one is given
        """
        def __enter__():
            pass
        def __exit__(*exc_info):
            pass

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Oct 8, 2010

    That is what we are using now, but I think a contextlib.null() would be useful to others, i.e. that its use is a useful idiom to adopt. Specifically I would like to discourage the "duplicated code" idiom from the report, which I've seen all too often.

    The "closing" constructor is also trivial to define, but it's there for convenience and to promote the use of with statement over try/finally boilerplate. The same goes here: you don't miss the null context manager when you don't have it; you invent other solutions. But when it's already available, it's an elegant pattern. In my experience, if they have to define it to get it, most people won't bother with the pattern and will retain less elegant solutions.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Oct 8, 2010

    Actually, the singleton idea isn't a bad one, but I'd go one step further and skip the factory function as well. So change that to be:

    class NullContext():
       ... # as per nullcontext in my last message
    
    nullcontext = NullContext()

    (with the example in the docstring adjusted accordingly)

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Oct 8, 2010

    If you can supply a full patch before the end of the month, we should be able to get this in for 3.2beta1 (currently scheduled for 31 October)

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Oct 8, 2010

    I considered using a variable, but I went with the factory function for two reasons: consistency with the rest of contextlib, and equivalence to the contextmanager-based implementation.

    Another reason is that it leaves the option of adding optional parameters at a later point. I know, optional parameters aren't likely for a "null" constructor, but still... it somehow didn't feel right to relinquish control.

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Oct 9, 2010

    Here is a more complete patch that includes input from Nick, as well as the patch to test_contextlib.py and the documentation.

    For now I've retained the function-returning-singleton approach for consistency and future extensibility.

    @ncoghlan ncoghlan self-assigned this Oct 10, 2010
    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Oct 12, 2010

    Is there anything else I need to do to have the patch reviewed and applied?

    I am in no hurry since we're still using 2.x, I'd just like to know if more needs to be done on my part to move the issue forward. My last Python patch was accepted quite some years ago, so I'm not closely familiar with the current approval process.

    @bitdancer
    Copy link
    Member

    @bitdancer bitdancer commented Oct 12, 2010

    Unless Nick has further feedback I think you've done all you need to, thanks.

    @giampaolo
    Copy link
    Contributor

    @giampaolo giampaolo commented Oct 12, 2010

    I'm with Antoine. Why not just do this in the context function itself?
    I think it's more explicit and easier than reading the doc to figure out what nullcontext is supposed to do:

    from contextlib import contextmanager
    
    CONDITION = False
    
    @contextmanager
    def transaction():
        if not CONDITION:
            yield None
        else:
            yield ...
    
    with transaction() as x:
        ...

    @bitdancer
    Copy link
    Member

    @bitdancer bitdancer commented Oct 12, 2010

    Because hardcoding a particular condition into a context manager is less flexible? (I'm +0 on this thing myself, by the way.)

    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Oct 12, 2010

    Are you sure that this is useful enough to warrant inclusion in the standard lib? I don't know of anyone else who has used the same idiom. It seems crufty to me -- something that adds weight (mental burden and maintenance effort) without adding much value. I don't know that anyone actually needs this.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Oct 13, 2010

    To me, this is more a matter of conceptual completeness than one of practical utility (ala fractions.Fraction). That said, I *have* personally encountered the "I only sometimes want to wrap this code in a CM" situation, so it isn't completely impractical, either. Those two factors are enough to reach my threshold for it being worthwhile to declare "one obvious way to do it" through the contextlib module.

    There is a possible alternative approach that may be more intuitive to use and read than nullcontext() though:

    @contextmanager
    def optional_cm(cm, *, use_cm=True): # See naming note below
        if cm is None or not use_cm:
            yield
        else:
            with cm:
                yield

    The OP's original example would then look like:

    with optional_cm(transaction):
        ...

    I suspect readers would find it far easier to remember what optional_cm does than to learn to recognise the "or nullcontext()" idiom. It also plays better with nested context managers:

    with optional_cm(sync_lock), optional_cm(db_transaction), \
         open(fname) as f:
        ...

    Naming Note: I nearly suggested "optional_context" as a name for this, but realised that would be subtly misleading (suggesting PEP-377 style functionality that potentially skipped the statement body, rather than the intended semantics of skipping use of the CM)

    @birkenfeld
    Copy link
    Member

    @birkenfeld birkenfeld commented Oct 14, 2010

    I like your latest suggestion, except for the name. Given that we also have the (quite generic) "closing", what about just "optional"?

    @rhettinger
    Copy link
    Contributor

    @rhettinger rhettinger commented Oct 14, 2010

    To me, this is more a matter of conceptual completeness
    than one of practical utility ...

    Nick, you don't seem to be truly sold on the need.
    I'm -1 on adding this. It's basically cruft. If
    it were published as an ASPN recipe, its uptake
    would be nearly zero.

    We need to focus on real problems in the standard
    library and provide solid solutions. If weight
    gets added to the standard lib, it needs to be
    selective.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Oct 24, 2010

    I find Raymond's perspective persuasive in this case. Feel free to post either the original idea or my later variant as an ASPN cookbook recipe. (you could actually combine the two, and use NullContext as an implementation detail of an optional_cm() function)

    @AlexanderJones
    Copy link
    Mannequin

    @AlexanderJones AlexanderJones mannequin commented Feb 8, 2012

    Not having this as a standard idiom makes it very tempting to just do copy-paste coding as in hniksic's example. Who likes to invent their own library for generic language-supporting idioms?

    What about an alternative of giving NoneType empty enter and exit methods? So instead of a 'null' CM you can just use "with None"?

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Feb 10, 2012

    FWIW, it's likely I'll be adding contextlib.ContextStack (see [1]) for 3.3. While it's far from the primary use case, that API also serves as a "no-op" context manager (if you never register any contexts or callbacks, the __exit__ impl does nothing).

    [1] http://contextlib2.readthedocs.org/en/latest/index.html#contextlib2.ContextStack

    @AlexanderJones
    Copy link
    Mannequin

    @AlexanderJones AlexanderJones mannequin commented Feb 10, 2012

    That's very reassuring. Thanks, Nick!

    @DLitz
    Copy link
    Mannequin

    @DLitz DLitz mannequin commented Nov 15, 2012

    After seeing a context manager named like "TempfileIfNeeded(..., cond)", whole sole purpose is to handle the conditional case, I'm firmly +1 on this proposal.

    It's much easier to just read "with Tempfile() if cond else nullcontext():" than to read through another level of indirection every time someone wanted some conditional logic on a context manager.

    Is there any chance that this issue could be reopened?

    Perhaps a more elegant solution would be to modify the "with" statement so that any object can be given to it (then we could just use None directly), but I suspect that would be a tad more controversial. ;)

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 16, 2012

    No, an empty ExitStack() instance already works fine as a no-op context manager in 3.3: http://docs.python.org/3/library/contextlib#simplifying-support-for-single-optional-context-managers

    We're not going to add a dedicated one under a different name.

    @MartinBlais
    Copy link
    Mannequin

    @MartinBlais MartinBlais mannequin commented Nov 18, 2016

    I've been looking for this today; I would have used it.

    Instead of an obvious (and explicit) null context manager, I had to read through this entire thread to eventually find out that I can use something called ExitStack(), which is designed for another use case.

    When many users have to replicate the same boilerplate code time and time again, it's not cruft, it's just something that ought to be part of the stdlib. There are a number of such cases in the stdlib. I think nullcontext should be part of the included batteries Python aims to provide.

    @giampaolo
    Copy link
    Contributor

    @giampaolo giampaolo commented Nov 18, 2016

    ExitStack() already covers the "null ctx mgr" use case described in the first message. Original example:

    with transaction or contextlib.null():
        ...

    By using ExitStack:

    with transaction or ExitStack():
        ...

    You can push this further and do this, which is even more flexible:

    with ExitStack() as stack:
        if condition:
            stack.enter_context(transaction)
        ...

    So ExitStack really is better than the original proposal which could have made sense 6 years ago but not anymore.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 19, 2016

    The problem Martin is referring to is the SEO one, which is that the top link when searching for either "null context manager python" or "no-op context manager python" is this thread, rather than the "Use ExitStack for that" recipe in the docs: https://docs.python.org/3/library/contextlib.html#simplifying-support-for-single-optional-context-managers

    We unfortunately have exactly zero SEO experts working on the CPython documentation, so even when we provide specific recipes in the docs for solving particular problems, they aren't always easy for people to find.

    I've at least added the "use contextlib.ExitStack()" note to the issue title here, so folks can find that without having to read through the whole comment thread.

    @ncoghlan ncoghlan changed the title Add a "no-op" (null) context manager to contextlib Add a "no-op" (null) context manager to contextlib (Rejected: use contextlib.ExitStack()) Nov 19, 2016
    @blais
    Copy link
    Mannequin

    @blais blais mannequin commented Nov 19, 2016

    Adding nullcontext = ExitStack in the source file would solve this problem in a single line of code.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 19, 2016

    No, it wouldn't, as ExitStack() does far more than merely implement a null context.

    It would be like adding "nulliterable = ()" as a builtin, rather than just telling people "If you need a null iterable, use an empty tuple".

    @blais
    Copy link
    Mannequin

    @blais blais mannequin commented Nov 19, 2016

    Well that just echoes exactly what I originally thought, but somebody else said it was not needed because ExitStack already exists and could be used for that purpose.

    If this were at work and/or it were all just to me, I'd just implement a brand new nullcontext and move on.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 20, 2016

    Unfortunately, the redundancy doesn't buy enough to justify the permanent documentation and style guide cost of providing two ways to do exactly the same thing.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 23, 2016

    It turns out that there's a variant on the "null context manager" idea that may *not* be redundant with ExitStack(), and hence could potentially counter the current rationale for not adding one.

    Specifically, it relates to context managers like click.progressbar() that are designed to be used with an "as" clause:

        with click.progressbar(iterable) as myiter:
            for item in myiter:
                ...

    At the moment, making that optional is a bit messy, since you need to do something like:

        with click.progressbar(iterable) as myiter:
            if not show_progress:
                myiter = iterable # Don't use the special iterator
            for item in myiter:
                ...

    or:

        with ExitStack() as stack:
            if show_progress:
                myiter = stack.enter_context(click.progressbar(iterable))
            else:
                myiter = iter(iterable)
            for item in myiter:
                ...

    or:

        @contextmanager
        def maybe_show_progress(iterable, show_progress)
            if show_progress:
                with click.progressbar(iterable) as myiter:
                    yield myiter
            else:
                yield iter(iterable)
    
        with maybe_show_progress(iterable, show_progress) as myiter:
            for item in myiter:
                ...

    The problem is that there's no easy way to say "return *this* value from __enter__, but otherwise don't do anything special".

    With a suitably defined NullContext, that last approach could instead look more like:

        if show_progress:
           ctx = click.progressbar(iterable)
        else:
           ctx = NullContext(iter(iterable))
    
        with ctx as myiter:
            for item in myiter:
                ...

    @albertz
    Copy link
    Mannequin

    @albertz albertz mannequin commented Nov 9, 2017

    Note that this indeed seems confusing. I just found this thread by search for a null context manager. Because I found that in TensorFlow they introduced _NullContextmanager in their code and I wondered that this is not provided by the Python stdlib.

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Nov 9, 2017

    For what it's worth, we are still using our own null context manager function in critical code. We tend to avoid contextlib.ExitStack() for two reasons:

    1. it is not immediately clear from looking at the code what ExitStack() means. (Unlike the "contextmanager" decorator, ExitStack is unfamiliar to most developers.)

    2. ExitStack's __init__ and __exit__ incur a non-negligible overhead compared to a true do-nothing context manager.

    It doesn't surprise me that projects like Tensor Flow introduce their own versions of this decorator. Having said that, I can also understand why it wasn't added. It is certainly possible to live without it, and ExitStack() is a more than acceptable replacement for casual use.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 10, 2017

    Reopening this based on several years of additional experience with context managers since I wrote https://bugs.python.org/issue10049#msg119514 when originally closing it.

    The version I'm now interested in adding is the one from https://bugs.python.org/issue10049#msg281556 - rather than being completely without behaviour, the null context manager should accept the value to be returned from the call to __enter__ as an optional constructor parameter (defaulting to None). That allows even context managers that return a value from __enter__ to be made optional in a relatively obvious way that doesn't involve fundamentally rearranging the code.

    I think the overhead argument against the use of ExitStack() for this purpose also has merit (so I'd be curious to see relative performance numbers collected with perf), but it's not my main motive for changing my mind.

    @ncoghlan ncoghlan reopened this Nov 10, 2017
    @ncoghlan ncoghlan changed the title Add a "no-op" (null) context manager to contextlib (Rejected: use contextlib.ExitStack()) Add a "no-op" (null) context manager to contextlib Nov 10, 2017
    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 10, 2017

    Reverting to "Needs patch", as the currently attached patch is for the "No behaviour" variant that always returns None from __enter__().

    (hniksic, would you still be willing to sign the Python CLA? If so, then your patch could be used as the basis for an updated implementation. Otherwise I'd advise anyone working on this to start from scratch)

    @hniksic
    Copy link
    Mannequin Author

    @hniksic hniksic mannequin commented Nov 15, 2017

    I am of course willing to sign the CLA (please send further instructions via email), although I don't know how useful my original patch is, given that it caches the null context manager.

    @Jesse-Bakker Jesse-Bakker mannequin added the 3.8 label Nov 19, 2017
    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 23, 2017

    New changeset 0784a2e by Nick Coghlan (Jesse-Bakker) in branch 'master':
    bpo-10049: Add a "no-op" (null) context manager to contextlib (GH-4464)
    0784a2e

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Nov 23, 2017

    Thanks to Jesse Bakker for the PR implementing this for 3.7!

    @ncoghlan ncoghlan added the 3.7 label Nov 23, 2017
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 3.8 stdlib type-feature
    Projects
    None yet
    Development

    No branches or pull requests

    9 participants