Decide how to backport 3.7 async context management support

[ncoghlan]

Python 3.7 added async/await support to the contextlib module (e.g. asyncontextmanager and AsyncExitStack), and then Python 3.10 added more (e.g. aclosing).

This raised the question of how to backport those features to 3.6+ (the first version with support for yield inside async def).

The answer turned out to be "procrastinate on the question for 4+ years, so it became feasible to simply drop support for Python 3.5 and earlier, and use the async generator code as-is".

(Issue description updated Jun 2021 to describe what actually happened, rather than the more complicated alternatives I was considering to allow adding the new async features without dropping support for Python 3.5 and earlier)

[sashgorokhov]

Can't imagine how you can manage this to work on 3.5 without yield support inside coroutines. I'll subscribe to this thread to see any useful ideas for that.

[sashgorokhov]

But supporting at least 3.6 would be great. I am already using personal package for this, and could provide a PR for asynccontextmanager

[JelleZijlstra]

Like Nick, I'd prefer keeping a single package. Having fewer packages with similar purposes makes it easier for people to decide what package to install.

The implementation would probably involve making contextlib2 into a package, organized something like this:

• _contextlib.py has all the current code
• _contextlib_async.py has @asynccontextmanager
• __init__.py does from _contextlib import * and if sys.version >= (3, 6): from _contextlib_async import asynccontextmanager. Thus, _contextlib_async would only be imported on Python versions where it's syntactically valid.

Yes, this probably can't be backported to 3.5 (maybe using https://github.com/njsmith/async_generator would work?). However, other future async enhancements to contextmanager, like AsyncExitStack, could be backported to 3.5 (and even earlier if you want to use yield from).

[sashgorokhov]

Using another libraries with their own logic ( yield_ ) will make it a pain in python 3.7 to migrate from this package to a standart one. Also, it won't be a clear solution for end-users

[ncoghlan]

For now, I've reworded the issue to just target 3.6+. However, we can keep the possibility of supporting earlier versions in mind as we consider possible solutions.

• the first option would be something like @JelleZijlstra suggested, only with the "minimum compatible version" files named after the earliest version that can import them (_contextlib_needs35.py, _contextlib_needs36.py, etc). I think this is the cleanest approach from an install time and runtime perspective, but the trickiest from the point of view of actually merging changes from the stdlib into the backport (but, as noted, we don't need to do that very often)

• the second option that comes to mind would be to include full copies of the versions of interest (so contextlib2._contextlib_37.py would be an actual pre-release copy of the 3.7 version of the module). Aside from the inevitable code duplication, the main argument against this is that it wouldn't help with cases like AsyncExitStack, where that's likely to be compatible with 3.5, even though asynccontextmanager will need 3.6. That's enough of a downside that I don't think any simplification of future maintenance could make it worthwhile to accept as a limitation

[njsmith]

FWIW, I use a home grown version of this all the time on 3.5, via https://github.com/njsmith/async_generator. For example, trio.open_nursery is implemented this way. The asynccontextmanager code itself doesn't need to define async generators, so there's no trouble importing it on 3.5; it just depends on the user to provide something that quacks like an async generator function, and it's up to them to decide if they want to use async_generator or whatever to make that function. (Why do I care about 3.5 support? PyPy.)

This code is sometimes a bit annoying to maintain, so I'd be happy to delegate to contextlib2 :-).

OTOH there might be a few challenges. The homegrown changes in my version are:

• Nice error message if someone accidentally uses with when they meant async with: Clearer error message if async context manager used synchronously python-trio/trio#212

• Adding some metadata so that functions returning async generators are recognizable: https://github.com/python-trio/trio/blob/e4c8ce7b81669f3e886c2ee434c7249aba7f55ed/trio/_util.py#L147-L148 (see bpo-30359, though here there's the additional wrinkle that AFAIK we don't yet have a way to express "async context manager" in __annotations__)

• (Not yet but someday) Right now, there's a race condition in signal handling when exiting from an async context manager: if control-C arrives just as we're entering the __aexit__ method, it causes it not to be run. In general trio protects from this using some complicated annotations, but I haven't bothered trying to solve this race condition yet, because even if I fixed it here then essentially the same race condition exists in the interpreter itself, so what's the point really. But if bpo-29988 got fixed then I would be motivated to figure out how to fix the part in the context manager as well, and it might be trickier if it lives in another library. Of course, contextmanager has the same problem, and I can probably solve it using some disgusting hack like noticing that all (async)contextmanager objects use the same code objects for their __(a)exit__ methods.

I don't think these are showstoppers, just figured put them on the radar.

[sorcio]

FYI, since the only piece I was missing from async_generator was an AsyncExitStack and since I only need Python 3.6, I packaged a "backport" for 3.5+ (more or less a verbatim copy from 3.7) here https://github.com/sorcio/async_exit_stack

I'd be happy to help with a proper backport.

[ncoghlan]

I think between @sorcio's async_exit_stack module and the simple passage of time, the idea of copying in the Python 3.7 version of the modules as contextlib2._contextlib_36_compat.py, moving the current code that implements stdlib APIs to contextlib2._contextlib_26_compat.py, and then populating contextlib2.__init__.py with code to choose the newest importable option and then add the extra contextlib2 specific backwards compatibility code is a lot more viable now than it was when I first opened this issue.

While there a few APIs it would be nice to extract and make universally available, that could be considered as a separate issue after the initial 3.6+ option was implemented.

[ncoghlan]

With @jayvdb submitting the synchronous enhancements as a separate PR (which I'm about to release as 0.6.0) that opens up a new alternative: what if contextlib2 0.7.0 were to just drop support for all versions prior to 3.6, leaving users of old Python versions on 0.6.0 indefinitely?

[graingert]

@ncoghlan does the Python 3.5 EOL change anything?

[ncoghlan]

@graingert It makes me even happier with the idea of 0.7.0 being 3.6+ only, which means it should be possible to just sync the latest version over from CPython 3.10 (modulo keeping the contextlib2 APIs that never graduated to the stdlib version)

[ncoghlan]

I've merged the changes from #29 (switch to CalVer, set the minimum version to 3.6), clearing the way for the Python 3.10 version to be sync'ed over. With the 2.x compatibility code gone, I'm hoping that will now be a lot simpler than it used to be.

[ncoghlan]

PR is up #32

CI mostly looks good, just need to check what's up with PyPy3

[ncoghlan]

As far as I can tell, the PyPy3 issue was a bug in the stdlib test suite (assuming the use of a refcounted GC): https://bugs.python.org/issue44515

CPython PR submitted at python/cpython#26910

[ncoghlan]

Bah, forgot to include the docs updates for the new APIs

[ncoghlan]

OK, docs are also updated now.