Support trio for structured async concurrency

[dhirschfeld]

trio is a Python library for structured async concurrency. As a trio user it would be great if async psycopg3 could support trio as well as asyncio.

The easiest way to support both async frameworks would be to use the anyio library.

[dvarrazzo]

I have separated the non-blocking part of the communication from the waiting. Using two different waiting function it is already possible to have a sync progress and an asyncio-based progress. There are some details at https://www.varrazzo.com/blog/2020/03/26/psycopg3-first-report/

Do you think trio could use the same code, only replacing the wait function?

[yogevyuval]

@dvarrazzo Really cool to see the beginning of psycopg3, it's long awaited.
Can you share some thoughts about how do you see this working along side SQLAlchemy, as well as the popular web frameworks? Flask, socket-io, etc...

[dvarrazzo]

@yogevyuval It's up to them to work asynchronously, we can only collaborate by not blocking. They have to call us asynchronously, being async is a different interface, as explained in the previous article.

[dhirschfeld]

@dvarrazzo - unfortunately I'm just a trio user so not qualified to comment on the implementation.

I don't want @njsmith or @agronholm to feel pressured to weigh in, especially with everything going on in the world, but I think my ping in gitter likely got lost in the noise - it's not a particularly good medium for keeping track of or searching for past issues.

[dvarrazzo]

@dhirschfeld I don't think a chat is a good medium to remember things, no. You can totally open a bug report to tell them of the library: If they are interested they can look into it further, otherwise they can dismiss it. I don't think that's putting them under undue pressure.

[agronholm]

I'm the author of AnyIO. I just had a cursory read of the blog post and would like to point out that add_reader() does not work on Windows (ProactorEventLoop). Also, if you intend to support more than just asyncio, then AnyIO is the most practical alternative. This means the code will have to get a bit more involved than just using some low level helpers.

[dvarrazzo]

Hi @agronholm,

the way I've designed the thing, the main loops interacting with the database (connection and querying) are independent from how you wait. I've done a few tests and it seems easy to create a wait function using anyio. Please see this gist.

I have a couple of observations:

• there are cases when we have to check if a file descriptor is ready either to read or to write, see PQflush documentation. With asyncio it doesn't seem possible to check that as it only offers functions wait_socket_readable() and _writable() methods.

• my test pass if the database connection is on unix socket (using PSYCOPG3_TEST_DSN="dbname=psycopg3_test" as connection string for the tests), but it explodes if connecting on a tcp socket (using PSYCOPG3_TEST_DSN="dbname=psycopg3_test host=localhost")

    async def wait_socket_writable(sock: socket.SocketType) -> None:
        check_cancelled()
        if _write_events.get(sock):
            raise ResourceBusyError('writing to') from None
    
        loop = get_running_loop()
        event = _write_events[sock] = asyncio.Event()
>       loop.add_writer(sock.fileno(), event.set)
E       AttributeError: 'int' object has no attribute 'fileno'

So I guess with a bit of work things can work.

[agronholm]

The problem with this is that IOCP on Windows does not support waiting for readability or writability. Instead it lets you wait for the completion of a read or write operation. For this reason, AnyIO 2.0 will not have the functionality for waiting for readability or writability.

[agronholm]

AnyIO 1.x does not work with ProactorEventLoop which is the default on Python 3.8 on Windows. AnyIO 2 does, but the cost was letting the aforementioned functions go.

[njsmith]

I'm pretty sure the need for readability/writability checking is baked into libpq, so it's not something psycopg3 can change.

Trio and uvloop do support waiting for readability/writability on Windows (via IOCP), so I guess anyio 2 could support that on most platforms (trio everywhere, asyncio on unix, asyncio on Windows with non-default event loops like SelectorEventLoop or uvloop). Not ideal, but better than not supporting psycopg3 at all.

[agronholm]

Not ideal, but better than not supporting psycopg3 at all.

I reluctantly agree.

[dvarrazzo]

Hello Trio people,

psycopg is due to release in a few days now. Would you like to check if it is compatible with trio, or if it can be made easily so (take a look at how AsyncConnection subclasses BaseConnection to have an idea how.

I am closing this ticket because it's no more psycopg2 matter. Please use psycopg/psycopg#29 or open new tickets in the psycopg 3 repos if needed

Cheers!