Fix alembic autogeneration and rename mismatching constraints #39032
+385
−30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:db-migrations
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
from
bd15707
to
c43f01b
Compare
airflow/migrations/versions/0142_2_10_0_update_missing_constraints.py
Outdated
Show resolved
Hide resolved
airflow/migrations/versions/0142_2_10_0_update_missing_constraints.py
Outdated
Show resolved
Hide resolved
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
2 times, most recently
from
e540be2
to
0d7c1f6
Compare
ephraimbuddy
added
the
full tests needed
uranusjr
approved these changes
Apr 23, 2024
|
The |
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
from
471afcb
to
bb7fd31
Compare
Should be fixed now |
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
2 times, most recently
from
7dcdcf8
to
aa441ce
Compare
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
from
aa441ce
to
ce50034
Compare
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
from
ce50034
to
5fd4b44
Compare
|
Previously, I added the |
jedcunningham
approved these changes
May 3, 2024
airflow/migrations/versions/0142_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py
Outdated
Show resolved
Hide resolved
airflow/migrations/versions/0142_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py
Show resolved
Hide resolved
airflow/migrations/versions/0142_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py
Show resolved
Hide resolved
The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful.
…ration-files an option in db commands
…use-migration-files an option in db commands
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
ephraimbuddy
force-pushed
the
fix-autogenerate
branch
from
4ef3b27
to
93357c3
Compare
RodrigoGanancia
pushed a commit
to RodrigoGanancia/airflow
that referenced
this pull request
May 10, 2024
…#39032) * Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
ephraimbuddy
added
type:bug-fix
utkarsharma2
pushed a commit
that referenced
this pull request
Jun 4, 2024
* Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> (cherry picked from commit 00f0969)
ephraimbuddy
added a commit
that referenced
this pull request
Jun 5, 2024
* Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> (cherry picked from commit 00f0969)
ephraimbuddy
added a commit
that referenced
this pull request
Jun 5, 2024
* Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> (cherry picked from commit 00f0969)
ephraimbuddy
added a commit
that referenced
this pull request
Jun 5, 2024
* Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> (cherry picked from commit 00f0969)
utkarsharma2
pushed a commit
that referenced
this pull request
Jun 5, 2024
* Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> (cherry picked from commit 00f0969)
ephraimbuddy
added a commit
that referenced
this pull request
Jun 5, 2024
* Fix alembic autogeneration and rename mismatching constraints The alembic autogeneration is not working as expected and the tests were detecting it because we use DBs created from the ORM to run tests. When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM. To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset. As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse. I also removed the `sqlite_sequence` table which is specific to sqlite and not needed for anything. An alternative would be to add `sqlite_autoincrement` to table args in the ORM and migration but this table is not that useful. * fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * fixup! fixup! fixup! Fix alembic autogeneration and rename mismatching constraints * Fix mysql, sqlite and issue with cascading deletes * fixup! Fix mysql, sqlite and issue with cascading deletes * Fix migration for mysql * Fix clear_number ORM server_default * Fix type change for mysql * Fix constraints for sqlite and move migration to 2.9.2 * Fix sqlite constraints update and ignore session_session_id_uq index * Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * fixup! Fix processor_subdir in the migration file for mysql and make use-migration-files an option in db commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> (cherry picked from commit 00f0969)
41 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The alembic autogeneration is not working as expected and the tests were not detecting it because we use DBs created from the ORM to run tests.
When a change is made in the ORM and the ORM is used to initialize the database for tests, the changes in the ORM will appear the same with what is in the migration file. To be sure that both match, we have to compare the database generated using the migration file to the database that could be created from the ORM.
To fix this, I added 'use_migration_file' arg to resetdb function and updated the db reset in conftest to use migration file during test db reset.
As part of this fix, I also updated mismatching constraint names. The update was done in the migration file instead of the ORM as I take the ORM as the source of truth. New airflow users create their DB from the ORM with the correct naming because we have a naming convention. Old airflow users would have to upgrade to use these names from ORM instead of the reverse.
I also removed the
sqlite_sequencetable which is specific to sqlite and not needed for anything. An alternative would be to addsqlite_autoincrementto table args in the ORM and migration but this table is not that useful.