Add context manager to temporarily disable GC

[rhettinger]

BPO	31356
Nosy	@rhettinger, @gpshead, @ncoghlan, @pitrou, @ned-deily, @pmp-p, @ericsnowcurrently, @serhiy-storchaka, @1st1, @MojoVampire, @YoSTEALTH, @bharel, @lisroach, @pablogsal, @sweeneyde, @Jongy
PRs	bpo-31356: Add context manager to temporarily disable GC #4224 bpo-31356: Fixing PyErr_WarnEx might error out. #5456 bpo-31356: Fix multiple errors in gc.ensure_disabled() #5462 Revert "bpo-31356: Add context manager to temporarily disable GC #4224 #5495 [3.7] Revert "bpo-31356: Add context manager to temporarily disable GC GH-5495 #5496

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-09-05.23:47:23.956>
labels = ['interpreter-core', '3.8', 'type-feature', 'library']
title = 'Add context manager to temporarily disable GC'
updated_at = <Date 2021-05-01.15:07:54.862>
user = 'https://github.com/rhettinger'

bugs.python.org fields:

activity = <Date 2021-05-01.15:07:54.862>
actor = 'Yonatan Goldschmidt'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2017-09-05.23:47:23.956>
creator = 'rhettinger'
dependencies = []
files = []
hgrepos = []
issue_num = 31356
keywords = ['patch']
message_count = 52.0
messages = ['301407', '301633', '301636', '301644', '301645', '301710', '301720', '302903', '305402', '306053', '306054', '306071', '306072', '306076', '311167', '311300', '311305', '311312', '311331', '311337', '311340', '311342', '311343', '311344', '311347', '311348', '311350', '311351', '311352', '311356', '311357', '311360', '311362', '311363', '311366', '311372', '311396', '311397', '311402', '311444', '311491', '311494', '311495', '311508', '311512', '311513', '311514', '311566', '311635', '311646', '385213', '385214']
nosy_count = 16.0
nosy_names = ['rhettinger', 'gregory.p.smith', 'ncoghlan', 'pitrou', 'ned.deily', 'pmpp', 'eric.snow', 'serhiy.storchaka', 'yselivanov', 'josh.r', 'YoSTEALTH', 'bar.harel', 'lisroach', 'pablogsal', 'Dennis Sweeney', 'Yonatan Goldschmidt']
pr_nums = ['4224', '5456', '5462', '5495', '5496']
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31356'
versions = ['Python 3.8']

[rhettinger]

Used like this:

    with gc_disabled():
        run_some_timing()

    with gc_disabled():
        # do_something_that_has_real_time_guarantees
        # such as a pair trade, robotic braking, etc

This would be implemented in C for atomicity and speed. It would be roughly equivalent to:

    @contextmanager
    def gc_disabled():
        if gc.isenabled():
            gc.disable()
            yield
            gc.enable()
        else:
            yield

[ncoghlan]

+1 from me for the general idea. As far as where to put it goes, I think the gc module would be the most appropriate home.

[serhiy-storchaka]

Note that threads can break this. Even without calling gc.enable() explicitly, "with gc_disabled()" in different thread can enable GC inside other "with gc_disabled()" block.

[ncoghlan]

Yes, this will be in the same category as the standard stream redirection context managers - multi-threaded applications either won't be able to use it, or else will need to manage access to it somehow.

[gpshead]

I believe Raymond is aware of the thread issue. We discussed it.

If gc.disable() would return the previous state of the gc instead of None and an API to enable based on a passed in bool, both of which are written in C and executed with the GIL (or merely another lock protecting changing the gc state rather than the GIL) held and this Python would works with multiple threads all fighting to control the gc state:

@contextmanager
def gc_disabled():
  previous_enable_state = gc.disable()
  yield
  gc.set_enable(previous_enable_state)

But we don't need to modify gc.disable's return value or add a set_enable() if gc_disabled() itself is not implemented in Python. That's up to the implementer.

[MojoVampire]

I'd be -1 on this without a demonstrated broad need for this in at least some context outside of microbenchmarking utilities (which presumably have already implemented similar stuff).

If a minimum bar for applicability isn't applied, we'll end up with dozens of these special purpose managers justified by the last limited applicability one. Stuff like contextlib.closing is justifiable (though I wish it allowed you to replace the method name called, so it could call terminate, kill, release, what-have-you) since cleanup close is so common, but disabling and reenabling gc is usually for timing purposes, and there aren't *that* many tools that need it.

It doesn't seem all that useful for real time purposes either; sure, it disables gc, but it doesn't disable other forms of implicit non-local code execution that are surprisingly hard to predict (e.g. object destruction, including __del__, for non-cyclic cases is going to depend on whether the stuff being decref-ed is still owned outside the block). Disabling GC means a lot in Java, because *all* cleanup is GC; in Python, it's just cycle collection, so you're not giving much in the way of guarantees, as the code in question has to be written with a *lot* of assumptions that Python usually can't support (it's hardly a realtime language).

Seems like if you want a speed up and reliability for this case (and all other context managers intended to be low overhead), it would be better to move parts of contextlib to the C layer (e.g. contextmanager and the classes it's implemented in terms of), so the simple and correct throwaway implementations for arbitrary custom use cases are fast enough; atomicity clearly doesn't actually matter here (it can't be made atomic in any meaningful sense given the lack of thread safety), so any real problem with the provided implementations (assuming they had try/finally added to make them robust) is overhead, not atomicity; the code posted so far would be fine (after adding try/finally) without the speed issues.

[ncoghlan]

This is being proposed for the gc module, not contextlib, so I'm not sure how the previous comment applies.

This is just a convenience API for folks already doing this kind of manipulation themselves.

[rhettinger]

Assigning this to Lisa for a C implementation, docs, and tests.