Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disabling prepared statements entirely with SQLAlchemy? #1041

Closed
interrogator opened this issue Jun 2, 2023 · 1 comment · Fixed by #1065
Closed

Disabling prepared statements entirely with SQLAlchemy? #1041

interrogator opened this issue Jun 2, 2023 · 1 comment · Fixed by #1065

Comments

@interrogator
Copy link

I have some SQL strings with multiple commands (i.e. INSERT a; INSERT B;), which cannot be made into prepared statements.

When using SQLAlchemy, it seems like asyncpg is always called in such a way that prepared statements are always generated.

There is an issue on a related subject, where the prepared_statement_cache_size option can be used to avoid cacheing the statements. But this setting does not stop them from being used entirely. Most of the above issue is only about changing the way such statements are named/identified/stored. Even with this set to 0, the statements are still created and called.

The only way I can find to get around this problem is to call asyncpg more directly:

engine = create_async_engine()
async with engine.begin() as conn:
    raw = await conn.get_raw_connection()
    await raw.cursor()._connection.execute(script)

Is there any nicer solution for this? I'd like to skip get_raw_connection, .cursor() and _connection, and use SQLAlchemy more as intended, but without generating any prepared statements.

Also, a somewhat related question, does SQLAlchemy provide a way of accessing asyncpg's copy_to_table other than the above? I can't find much documentation about this, nor anything specific to COPY in the source.

@elprans
Copy link
Member

elprans commented Aug 15, 2023

But this setting does not stop them from being used entirely

Prepared statements are an integral part of PostgreSQL extended query protocol which asyncpg uses. That said, I'm not sure why you would want to disable those as that flow should work just fine with pgbouncer. Can you give a more concrete example of a problem you're seeing?

elprans added a commit that referenced this issue Aug 16, 2023
It appears that PgBouncer's `transaction` pooling mode does not consider
implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`,
`Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by
PgBouncer as a transaction boundary and it will happily send the
following `Bind` / `Execute` messages to a different backend process.

This makes it so that when `statement_cache_size` is set to `0`, asyncpg
assumes a pessimistic stance on prepared statement persistence and does
not rely on them even in implicit transactions.  The above message
sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`,
`Execute`, `Sync`.

This obviously has negative performance impact due to the extraneous
`Parse`.

Fixes: #1058
Fixes: #1041
elprans added a commit that referenced this issue Aug 16, 2023
It appears that PgBouncer's `transaction` pooling mode does not consider
implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`,
`Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by
PgBouncer as a transaction boundary and it will happily send the
following `Bind` / `Execute` messages to a different backend process.

This makes it so that when `statement_cache_size` is set to `0`, asyncpg
assumes a pessimistic stance on prepared statement persistence and does
not rely on them even in implicit transactions.  The above message
sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`,
`Execute`, `Sync`.

This obviously has negative performance impact due to the extraneous
`Parse`.

Fixes: #1058
Fixes: #1041
elprans added a commit that referenced this issue Aug 17, 2023
…1065)

It appears that PgBouncer's `transaction` pooling mode does not consider
implicit transactions properly, and so in a [`Parse`, `Flush`, `Bind`,
`Execute`, `Sync`] sequence, `Flush` would be (incorrectly) considered by
PgBouncer as a transaction boundary and it will happily send the
following `Bind` / `Execute` messages to a different backend process.

This makes it so that when `statement_cache_size` is set to `0`, asyncpg
assumes a pessimistic stance on prepared statement persistence and does
not rely on them even in implicit transactions.  The above message
sequence thus becomes `Parse`, `Flush`, `Parse` (a second time), `Bind`,
`Execute`, `Sync`.

This obviously has negative performance impact due to the extraneous
`Parse`.

Fixes: #1058
Fixes: #1041
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants