-
Notifications
You must be signed in to change notification settings - Fork 398
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
asyncpg uses prepared statements despite they're disabled #1058
Comments
Unfortunately, I'm unable to reproduce the error using your script (I tried on Python 3.10 and 3.11). |
hi @elprans - this error reproduces if you run it against pgbouncer in "transaction" mode, here is a config file I used for pgbouncer to reproduce this:
also, can we look at the error message that asyncpg is emitting, which is also in the FAQ - it suggests using "statement_cache_size = 0" in order to mitigate the issue, however our test case here shows that this is not sufficient. if you were able to reproduce this, would it be considered a bug, or is the whole notion of using "transaction" mode in pgbouncer something that asyncpg should say is explicitly not supported? |
OK, thanks, can repro now.
It's technically a bug. Or, rather, insufficiently strong mitigation. For now a solution would be to force another That said, |
It appears that PgBouncer's `transaction` pooling mode does not consider implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`, `Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by PgBouncer as a transaction boundary and it will happily send the following `Bind` / `Execute` messages to a different backend process. This makes it so that when `statement_cache_size` is set to `0`, asyncpg assumes a pessimistic stance on prepared statement persistence and does not rely on them even in implicit transactions. The above message sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`, `Execute`, `Sync`. This obviously has negative performance impact due to the extraneous `Parse`. Fixes: #1058 Fixes: #1041
when used in this mode (and as seen in the test case) we are giving our prepared statements names and they run on pgbouncer without issue. the issue here is that a totally unrelated statement that is using We can "fix" this issue on our end by using additionally it seems, though maybe you can clarify, that it would be quite wasteful to do |
This is surprising to me. This is from the PgBouncer FAQ: "How to use prepared statements with transaction pooling? To make prepared statements work in this mode would need PgBouncer to keep track of them internally, which it does not do. So the only way to keep using PgBouncer in this mode is to disable prepared statements in the client."
|
Here's an illustration of what PgBouncer sends to Postgres in the above script (I modified
Note two things:
There is no reason why this wouldn't happen with explicit named prepared statements. |
It appears that PgBouncer's `transaction` pooling mode does not consider implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`, `Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by PgBouncer as a transaction boundary and it will happily send the following `Bind` / `Execute` messages to a different backend process. This makes it so that when `statement_cache_size` is set to `0`, asyncpg assumes a pessimistic stance on prepared statement persistence and does not rely on them even in implicit transactions. The above message sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`, `Execute`, `Sync`. This obviously has negative performance impact due to the extraneous `Parse`. Fixes: #1058 Fixes: #1041
So it's weird they say that, because I think I understand the problem now and that FAQ is not really accurate. Ive googled a bit for other takes on this issue, and the issue really is nothing more than "transaction" mode is going to give you a different connection for each command, as you've illustrated in your example above. So what the FAQ does not say is that prepared statements won't work, if you did not actually begin a transaction. Because that's what "transaction" mode means; pgbouncer will give you the same connection for all commands if you open a transaction and work inside that. So we can make the above test case work just like this: async def main():
conn = await asyncpg.connect(
user="scott",
password="tiger",
host="127.0.0.1",
database="test",
port=6432,
statement_cache_size=0,
)
async with conn.transaction():
pps = await conn.prepare("select 1", name=f'__asyncpg_{uuid.uuid4()}__')
rows = await pps.fetch()
# del or not, doesn't matter
del pps
(await conn.fetchrow(";")) the other part of it is that the prepared statement we use explicitly has a name using the feature added by #837, that is asyncpg otherwise will name it something that might be already present on a different PG connection that nonetheless to asyncpg looks like the same connection, as pgbouncer switches connections around behind the scenes. This itself could also be mitigated by using server_reset_query_always , while potentially though not necessarily setting server_reset_query to use DEALLOCATE ALL instead of DISCARD ALL. Again pgbouncer's docs here are misleading when they use terminology like "This setting is for working around broken setups that run applications that use session features over a transaction-pooled PgBouncer." So what's happening in SQLAlchemy is that we've wrapped asyncpg behind a layer that provides pep-249 compatibility, which includes, all SQL commands implicitly auto-begin a transaction. That's why things work for us. This particular "SELECT ;" thing is a liveness ping, that occurs outside of any transaction.....on the asyncpg driver. On other drivers like pg8000, the DBAPI always uses a transaction. So. Great news for asyncpg, I can fix this on my end just by making sure our "ping" is run inside of a transaction. clearly asyncpg can't do much here, would be nice if pgbouncer had less opinionated documentation that is not so confusing to everyone. |
Yes, except they completely overlook that implicit transactions exist and break those.
I think I'm still going to merge the "redo the Parse if statement cache is off" PR. |
sounds great! |
…1065) It appears that PgBouncer's `transaction` pooling mode does not consider implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`, `Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by PgBouncer as a transaction boundary and it will happily send the following `Bind` / `Execute` messages to a different backend process. This makes it so that when `statement_cache_size` is set to `0`, asyncpg assumes a pessimistic stance on prepared statement persistence and does not rely on them even in implicit transactions. The above message sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`, `Execute`, `Sync`. This obviously has negative performance impact due to the extraneous `Parse`. Fixes: #1058 Fixes: #1041
Hi all - is there any workaround for this? I'm using SQLModel/sqlalchemy on Supabase, but this is blocking moving to async. Setting the cache to 0 doesn't seem to work, perhaps this is incorrect? (The error only happens on the second query).
async_engine = create_async_engine(
async_connection_str,
pool_pre_ping=True,
pool_size=8,
connect_args={"statement_cache_size": 0}, # doesn't seem to work?
)
# first query is fine
async with AsyncSession(async_engine) as session:
result = await session.get(MyRow, "123")
# second query throws error
async with AsyncSession(async_engine) as session:
result = await session.get(MyRow, "123")
# sqlalchemy.exc.ProgrammingError: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError)
# <class 'asyncpg.exceptions.DuplicatePreparedStatementError'>:
# prepared statement "__asyncpg_stmt_1__" already exists edit: it's now started failing on the first query as well. |
Hi @nsbradford. I've had same error and fixed it by adding |
Thanks for the link! But now i get
Presumably because i'm on SQLAlchemy 1.4 instead of 2.0+. The legacy docs (https://docs.sqlalchemy.org/en/14/dialects/postgresql.html#module-sqlalchemy.dialects.postgresql.asyncpg) seem to suggest
However, this doesn't seem to work either :( still
|
@nsbradford |
@nsbradford in my project I added # sqlalchemy 1.4 example
from asyncpg import Connection
class UniquePreparedStatementConnection(Connection):
def _get_unique_id(self, prefix: str) -> str:
return f"__asyncpg_{prefix}_{uuid.uuid4()}__"
engine = create_async_engine(
url=url,
connect_args={
"connection_class": UniquePreparedStatementConnection,
"statement_cache_size": 0,
"prepared_statement_cache_size": 0,
},
)
# sqlalchemy 2.0 example
engine = create_async_engine(
url=url,
connect_args={
"prepared_statement_name_func": lambda: f"__asyncpg_{uuid.uuid4()}__",
"statement_cache_size": 0,
"prepared_statement_cache_size": 0,
},
) |
Started from sqlalchemy/sqlalchemy#10226.
asyncpg uses prepared statements despite they're disabled.
To reproduce (thanks @zzzeek for the script!):
Error
The text was updated successfully, but these errors were encountered: