Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A way to run non-transactional DDL commands / PG requires autocommit to modify a type ?! #123

Closed
sqlalchemy-bot opened this issue May 10, 2013 · 11 comments

Comments

@sqlalchemy-bot
Copy link

commented May 10, 2013

Migrated issue, originally created by Wichert Akkerman (@wichert)

Unfortunately not all DDL commands for PostgreSQL can be used in a transaction. I ran into this when trying to add a new value to an enum:

db=# BEGIN;
BEGIN
db=# ALTER TYPE article_type ADD VALUE 'unknown';
ERROR:  ALTER TYPE ... ADD cannot run inside a transaction block

I'm not sure what the best way to handle this is in alembic currently. Perhaps do op.execute('COMMIT') manually? Perhaps alembic needs an in-between-commit operation for this.

@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented May 10, 2013

Michael Bayer (@zzzeek) wrote:

Well, this is tricky because the BEGIN is emitted by psycopg2. You might need to just get a raw_connection() from "op.bind" and work with the psycopg2 cursor directly for that operation.

@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented May 11, 2013

Wichert Akkerman (@wichert) wrote:

Looks like psycopg2 doesn't do that here for some reason. This code ran without problems:

op.execute('COMMIT')  # See https://bitbucket.org/zzzeek/alembic/issue/123
op.execute('ALTER TYPE article_type ADD VALUE \'unknown\'')
@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented May 11, 2013

Michael Bayer (@zzzeek) wrote:

that's a little weird, some DBAPIs look at the SQL string to determine "auto-begin" but I didn't think psycopg2 was doing that. If you turn on statement logging in your postgresql database it'll reveal just what's going on with that.

@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented May 11, 2013

Michael Bayer (@zzzeek) wrote:

oh duh, because you said COMMIT, psycopg2 thought it was still in its own transaction block. I'm not sure I'd want to go further than illustrating the workaround here in the alembic docs, something like that. we'd need "database-specific notes".

@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented Jun 7, 2013

Changes by Michael Bayer (@zzzeek):

  • added labels: documentation
@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented Sep 23, 2014

Lucas Taylor (@ltvolks) wrote:

Just a note that you can also set the isolation level on the connection itself prior to executing the ALTER statement:

if not op.get_context().as_sql:
    connection = op.get_bind()
    connection.execution_options(isolation_level='AUTOCOMMIT')

op.execute('ALTER TYPE article_type ADD VALUE \'unknown\'')

@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented Dec 19, 2014

Wichert Akkerman (@wichert) wrote:

@ltvolks: I'm afraid that breaks for other reasons. I tried both your version and the simpler op.execute(...., {'isolation_level': 'AUTOCOMMIT'})), but in both cases psycopg will issue a ROLLBACK which blasts away any earlier changes. This is the PostgreSQL statement log:

LOG:  statement: BEGIN
LOG:  statement: SELECT t.oid, typarray
	FROM pg_type t JOIN pg_namespace ns
	    ON typnamespace = ns.oid
	WHERE typname = 'hstore';
	
LOG:  statement: ROLLBACK
LOG:  statement: BEGIN
LOG:  statement: select version()
LOG:  statement: select current_schema()
LOG:  statement: show transaction isolation level
LOG:  statement: SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
LOG:  statement: SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
LOG:  statement: SELECT 'x' AS some_label
LOG:  statement: ROLLBACK
LOG:  statement: BEGIN
LOG:  statement: show standard_conforming_strings
LOG:  statement: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where n.nspname=current_schema() and relname='alembic_version'
LOG:  statement: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where n.nspname=current_schema() and relname='alembic_version'
LOG:  statement: 
	CREATE TABLE alembic_version (
		version_num VARCHAR(32) NOT NULL
	)
	
	
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: ROLLBACK
LOG:  statement: SET default_transaction_isolation TO DEFAULT
LOG:  statement: ALTER TYPE article_type ADD VALUE IF NOT EXISTS 'necklace'
LOG:  statement: INSERT INTO alembic_version (version_num) VALUES ('4bd730372bac')
ERROR:  relation "alembic_version" does not exist at character 13
STATEMENT:  INSERT INTO alembic_version (version_num) VALUES ('4bd730372bac')
LOG:  statement: SET default_transaction_isolation TO 'read committed'
@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented Dec 19, 2014

Lucas Taylor (@ltvolks) wrote:

@wichert: I believe that psycopg2 is issuing a ROLLBACK here because it detects an in-progress transaction. I'm guessing that when you change the isolation level, the driver needs to be in a known state so it issues the ROLLBACK.

If you can, try setting transaction_per_migration=True for the migration context. This is the setting I use and in the statement log below, we can see that there is no ROLLBACK after SHOW default_transaction_isolation, presumably because a transaction has not already begun. This aligns with your earlier comment.

If that isn't acceptable for your workflow, then you might need to isolate your migrations that need to be run outside of a transaction and step through the upgrades incrementally (alembic upgrade +1)

LOG:  statement: BEGIN
LOG:  statement: SELECT t.oid, typarray
    FROM pg_type t JOIN pg_namespace ns
        ON typnamespace = ns.oid
    WHERE typname = 'hstore';
    
LOG:  statement: ROLLBACK
LOG:  statement: BEGIN
LOG:  statement: select version()
LOG:  statement: select current_schema()
LOG:  statement: show transaction isolation level
LOG:  statement: SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
LOG:  statement: SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
LOG:  statement: SELECT 'x' AS some_label
LOG:  statement: ROLLBACK
LOG:  statement: BEGIN
LOG:  statement: show standard_conforming_strings
LOG:  statement: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where n.nspname=current_schema() and relname='alembic_version'
LOG:  statement: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where n.nspname=current_schema() and relname='alembic_version'
LOG:  statement: 
    CREATE TABLE alembic_version (
        version_num VARCHAR(32) NOT NULL
    )
    
    
LOG:  statement: COMMIT
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: SET default_transaction_isolation TO DEFAULT
LOG:  statement: ALTER TYPE article_type ADD VALUE IF NOT EXISTS 'necklace'
LOG:  statement: INSERT INTO alembic_version (version_num) VALUES ('4951454fe44c')
LOG:  statement: SET default_transaction_isolation TO 'read committed'

This log was generated with:

  • stock alembic env.py, with the addition of transaction_per_migration=True
  • No alembic_version table to begin
  • Pre-created ENUM article_type
  • Single migration revision containing:
def upgrade():
    if not op.get_context().as_sql:
        connection = op.get_bind()
        connection.execution_options(isolation_level='AUTOCOMMIT')

    op.execute("ALTER TYPE article_type ADD VALUE IF NOT EXISTS 'necklace'")

(Your log can be reproduced exactly with the default transaction_per_migration=False)

@sqlalchemy-bot

This comment has been minimized.

Copy link
Author

commented Jul 22, 2015

Josh Kupershmidt (@schmiddy) wrote:

@ltvolks I believe you need to remember to set the connection back into READ_COMMITTED mode (or whatever it was prior to being forced into AUTOCOMMIT) at the end of upgrade(), since there may be subsequent migrations after the current one, right? E.g.

def upgrade():
    connection = None
    if not op.get_context().as_sql:
        connection = op.get_bind()
        connection.execution_options(isolation_level='AUTOCOMMIT')

    op.execute("ALTER TYPE article_type ADD VALUE IF NOT EXISTS 'necklace'")

    if connection is not None:
        connection.execution_options(isolation_level='READ_COMMITTED')
@zzzeek zzzeek changed the title A way to run non-transactional DDL commands A way to run non-transactional DDL commands / PG requires autocommit to modify a type ?! Sep 17, 2019
@zzzeek

This comment has been minimized.

Copy link
Member

commented Sep 17, 2019

if I hope to implement migration operations for ENUM including autogenerate support, if PG won't even let us add elements to an enum without cancelling the transaction, that's going to be really painful

@sqla-tester

This comment has been minimized.

Copy link
Collaborator

commented Sep 17, 2019

Mike Bayer has proposed a fix for this issue in the master branch:

Add autocommit_block https://gerrit.sqlalchemy.org/1463

sqlalchemy-bot pushed a commit that referenced this issue Sep 17, 2019
Change-Id: Idee4a70b03a6c3939e77573b14658a67a07e6cea
Fixes: #123
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.