PyAsyncGenASend objects allocated from freelists may not have their finalizers called

[colesbury]

Bug report

CPython uses freelists to speed up allocation of certain frequently allocated types of objects. CPython also supports finalizers (i.e., tp_finalize) that are only called once, even if the object is resurrected by its finalizer. These two features do not work well together as currently implemented because we don't clear the _PyGC_PREV_MASK_FINALIZED bit when objects are allocated from free-lists.

As far as I can tell, this only affects PyAsyncGenASend objects -- I haven't seen other objects that are both allocated from free-lists and use tp_finalize.

The finalizer for PyAsyncGenASend (which may issue a warning), may not be called if the object is allocated from a free-list (and already finalized):

Test case

The test(False) call should issue a warning about unawaited "asend" coroutine. However, the presence of the test(true) call will suppress this warning (if uncommented) because it ensures that there is an already-finalized object in the free-list.

import asyncio

def main():
    loop = asyncio.new_event_loop()

    async def gen():
        yield 1

    async def test(do_await):
        g = gen()
        if do_await:
            r = await g.asend(None)
        else:
            g.asend(None)
        await g.aclose()

    # Uncommenting this line prevents the warning on the following call
    # due to the already finalized PyAsyncGenASend object in the free-list.
    # loop.run_until_complete(test(True))

    # This should warn!
    loop.run_until_complete(test(False))

    loop.close()


if __name__ == '__main__':
    main()

Linked PRs

• gh-113753: Clear finalized bit when allocating PyAsyncGenASend from free-list #113754

[gvanrossum]

Technically this isn't an asyncio issue (it's part of asynchronous generators, which have uses outside asyncio), but I'll allow it. :-)

I'm not familiar with the GC infrastructure -- is the solution as simple as arranging to clear the bit when allocating from the free list?

[colesbury]

@gvanrossum - yeah it could be as simple as adding a line like the following (possibly refactored into a function):

        _Py_AS_GC((PyObject *)o)->_gc_prev &= ~_PyGC_PREV_MASK_FINALIZED;

to where we allocate from the free-list:

cpython/Objects/genobject.c

Lines 1913 to 1917 in 99854ce

	if (state->asend_numfree) {
	state->asend_numfree--;
	o = state->asend_freelist[state->asend_numfree];
	_Py_NewReference((PyObject *)o);
	}

[gvanrossum]

Cool, make the PR and CC me.

[hugovk]

Triage: #113754 has been merged, can this be closed?

[gvanrossum]

@colesbury Is there any worth in backporting this?

[colesbury]

@hugovk, yes I think it can be closed. I went ahead and closed the issue.

@gvanrossum The finalizer for PyAsyncGenASend was only added in 3.13 so I don't think we need to backport this.