Fix #529 2 - "This Session's transaction has been rolled back"

[LAlbertalli]

This pull request fixes the second issue found on #529

Bug:

• A wrong foreign key has been set on the table columns. It creates a constraint between the column name and the datasource_name in the datasources table.

Fix:

• Create a migration to change the foreign key to datasource_name in columns -> datasource_names in datasources.

[landscape-bot]

Repository health decreased by 0.04% when pulling 29ffe03 on LAlbertalli:fix529-2 into 7d27692 on airbnb:master.

• 1 new problem was found (including 0 errors and 1 code smell).
• No problems were fixed.

[LAlbertalli]

Hi,

I noticed that the migrations fails on sqlite. The problem is linked to the fact that SQL Lite does not support alter tables and I have to use a batch alter instead:

    with op.batch_alter_table('columns') as batch_op:
        batch_op.drop_constraint('columns_ibfk_1', type_="foreignkey")
        batch_op.create_foreign_key('columns_ibfk_1', 'datasources', ['datasource_name'], ['datasource_name'])

The problem with this approach is that the sqlite schema doesn't have a name on the foreign key so there's no way I could drop it.

[xrmx]

@LAlbertalli since sqlite is the default db please add the batch context manager to keep caravel working out of the box.

[LAlbertalli]

@xrmx Please, read my full comment. This kind of migration is not supported in sqlite because the constraint as no name. Otherwise, I would have submitted it immediately. That's why I'm asking for the best approach.

There's a point to be made, though. Foreign keys are not on by default on sqlite, that's probably why none noticed the bug before. Could we modify the init migrations to have the correct foreign key and we conditionally skip the migration for sqlite? I don't like the idea because it will introduce an unpredictable behavior. On the other side, it is the only upgrade path I could foresee that doesn't require an overly complex setup.

I'm open to suggestion.

[xrmx]

@LAlbertalli does this help? http://alembic.readthedocs.io/en/latest/batch.html#dropping-unnamed-or-named-foreign-key-constraints

UPDATE: it looks it breaks mysql, see #466 which looks like the same issue as #529

[LAlbertalli]

@xrmx Yes, that was the same issue I found trying with the suggestion in the doc you linked.
Another bigger problem is that I'm naming directly the constraint, but that name seems to be autogenerated by MySQL. But I'm not sure the constraint name will be the same on PG that is the third supported database.

I need to do further research, this is why I was asking further suggestion from people more experienced than me in using SQLAlchemy/Alembic.

[x4base]

@xrmx Seems like it only works when you have created the foreign keys with naming_convention specified. But it's too late now, for those who has been using mysql, the foreign keys have been named according to the default name convention of MySQL. The only solution I have figured out so far is to try on every kind of databases and try to drop the key by trying through each of the default foreign key names.

[LAlbertalli]

@xrmx @x4base I see two options moving forward. Modifying the init migration to fix the error. I'm not happy with that but it should solve the issue for new install. People who are already using it, or they don't check foreign keys, or they have already fixed it.

The second option, that I actually like more, is to surround the migration with an exception handling code to skip sqlite and find the correct name for PGSQL.

To be honest, I will go for both actions (change init and use error handling) to cover all the possible issues.

Let me know what do you prefer. It will take me probably few days to provide the patch.

Bye
L.

[x4base]

I agree with droping the key by the pgsql. But i am concerned about whether we should cover more Sql dialects (e.g. mssql) that sqlalchemy supports or should we just assume that no one is using other dialects?

[LAlbertalli]

@x4base I was thinking of sqlite, MySQL, and pg because they are the databases listed in https://github.com/airbnb/caravel/blob/master/caravel/config.py#L35 as an example. But, yes, nothing forbid to use MS SQL or Oracle. If someone has ideas...

[LAlbertalli]

The last commit should work for every supported DB.

I've tested it against sqlite and partially tested against MySQL and it worked both upgrade and downgrade.

The "magic" is done by a function that uses the sqlalchemy reflection capability to detect the correct constraint name.

[coveralls]

Coverage remained the same at 82.108% when pulling 7308cf6 on LAlbertalli:fix529-2 into 087c47a on airbnb:master.

[coveralls]

Coverage remained the same at 82.108% when pulling 7308cf6 on LAlbertalli:fix529-2 into 087c47a on airbnb:master.

[xrmx]

On top of this should we probabily generalize this function so can be reused in other migrations?

[LAlbertalli]

I agree in general.
There's just a question of timing, I don't have much time to dedicate on this bug fix right now and I think this issue is pretty urgent (basically Druid doesn't work on any DB that support Foreign Keys).

[xrmx]

Sure, i can do it after this is in :)

[mistercrunch]

👍