queries are blocking on gevent after 3.1.4

[residentsummer]

I've encountered an issue with psycopg calls blocking execution of other greenlets when running under gevent. Initially, problem manifested itself in celery worker, but during a debugging session I've reproduced it with the demonstration script from issue #4 . Here is the output of running it with psycopg[c]==3.1.8 and gevent==22.10.2 (latest version ATM):

$ python3 ./test.py 
2023-03-21 16:12:32,165 start threads
2023-03-21 16:12:32,165 connecting 0
2023-03-21 16:12:32,166 connecting 1
2023-03-21 16:12:32,166 connecting 2
2023-03-21 16:12:32,178 querying 1
2023-03-21 16:12:34,183 queried 1
2023-03-21 16:12:34,184 querying 0
2023-03-21 16:12:36,191 queried 0
2023-03-21 16:12:36,194 querying 2
2023-03-21 16:12:38,205 queried 2
2023-03-21 16:12:38,206 end threads

After poking around in psycopg/waiting.py and using particular undocumented env var to set wait function to wait_select or wait_selector, I've observed expected behavior - queries were running concurrently. Choosing wait_c or unsetting the env var resulted in blocking wait. I've traced the issue back to commit 9465578 between 3.1.4 and 3.1.5 (checked on psycopg[c]==3.1.4 - no issue indeed).

I suppose that monkey-patching, that gevent does, is not working for functions from C modules. Is there anything that can be done to enable concurrent queries? What are benefits of wait_c over wait_select[or]?

Python 3.10.6 on Linux, aarch64
libpq-dev 14.7-0ubuntu0.22.04.1
libpq5:arm64 14.7-0ubuntu0.22.04.1

[dvarrazzo]

Yes, psycopg 3 is not greenlet friendly; it was as a side effect of using a Python wait loop, but since the wait loop has been optimised in C the monkey-patching doesn't work anymore. The benefit of wait_c is performance.

There are possibilities to use a different wait function, as you have seen. I still prefer that the default is performing for standard python (async or blocking) which exactly like it was for psycopg 2, which required specific wait callback for gevent interaction, provided by psycogreen.

For gevent it would be enough to use the wait_select function; alternatively we can introduce a gevent-specific function (but then I assume we'd need an eventlet one too?) Or we can let it do to psycogreen.

I am not familiar anymore with gevent development, I didn't expect to be still a thing. What is the best interface to give to the feature? How would it be to detect that gevent has been imported and use a gevent-friendly wait function?

[dvarrazzo]

Psycopg 3.1.14 reintroduced gevent compatibility.

[residentsummer]

@dvarrazzo Thank you for this change!