Skip to content

Add server default for map_index in Log table#23056

Merged
dstandish merged 3 commits into
apache:mainfrom
astronomer:add-server-default-for-map-index-in-log-table
Apr 19, 2022
Merged

Add server default for map_index in Log table#23056
dstandish merged 3 commits into
apache:mainfrom
astronomer:add-server-default-for-map-index-in-log-table

Conversation

@dstandish
Copy link
Copy Markdown
Contributor

@dstandish dstandish commented Apr 18, 2022

When logging CLI actions we insert a record into the Log table. But for 2.3 we add column map_index to Log, and if the Log model expects map_index to be there the insert will fail and a warning will be emitted.

We can avoid the error and warning by adding a server_default of NULL on map_index in Log. I choose NULL instead of -1 because generally speaking map_index doesn't make sense for Log tables.

@github-actions github-actions Bot added the full tests needed We need to run full set of tests for this PR to merge label Apr 18, 2022
@github-actions
Copy link
Copy Markdown
Contributor

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@uranusjr
Copy link
Copy Markdown
Member

I think this needs a change in the migration file as well?

@dstandish
Copy link
Copy Markdown
Contributor Author

I think this needs a change in the migration file as well?

@uranusjr in the migration file it's nullable which means essentially that the server default is NULL already. So I don't think we need to change anything more but let me know if you think there's something we need to do.

@uranusjr
Copy link
Copy Markdown
Member

in the migration file it's nullable which means essentially that the server default is NULL already.

But the definition in the class also declares nullable (in the sense that nullable is not False and True is the default). From what I understand, server_default is injected into ALTER TABLE when the table is being modified (or CREATE TABLE), and without adding this to a migration this would do nothing.

Instead of server_default perhaps we need to add self.map_index = None in __init__ instead.

@dstandish
Copy link
Copy Markdown
Contributor Author

From what I understand, server_default is injected into ALTER TABLE when the table is being modified (or CREATE TABLE), and without adding this to a migration this would do nothing.

So it does something -- namely, it seems to make it so that when it generates an insert, it doesn't include map_index as a column when it's not set. I learned of the behavior here. And I did test it locally. I don't think adding to migration would change the behavior (since NULL is already the server default, right?) but it's easy enough to do so I'll add that in a sec.

Instead of server_default perhaps we need to add self.map_index = None in __init__ instead.

I just tried this and it doesn't work. I suspect it already gets a None value since it doesn't get set in init.

You can test locally on main by taking a fresh db, dropping map_index from log table, and running db upgrade.

@dstandish dstandish force-pushed the add-server-default-for-map-index-in-log-table branch from 86fc312 to b4b646d Compare April 19, 2022 17:03
@ashb ashb force-pushed the add-server-default-for-map-index-in-log-table branch from b4b646d to b0326c8 Compare April 19, 2022 18:01
When logging CLI actions we insert a record into the Log table.  But for 2.3 we add column map_index to Log, and if the Log model expects map_index to be there the insert will fail and a warning will be emitted.

We can avoid the error and warning by adding a server_default of NULL on map_index in Log. I choose NULL instead of -1 because
generally speaking map_index doesn't make sense for Log tables.
@dstandish dstandish force-pushed the add-server-default-for-map-index-in-log-table branch from b0326c8 to 85f2a24 Compare April 19, 2022 18:13
@dstandish dstandish merged commit f471d4f into apache:main Apr 19, 2022
@dstandish dstandish deleted the add-server-default-for-map-index-in-log-table branch April 19, 2022 21:29
@jedcunningham jedcunningham added changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) area:dynamic-task-mapping AIP-42 labels Apr 25, 2022
@jedcunningham jedcunningham added this to the Airflow 2.3.0 milestone Apr 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dynamic-task-mapping AIP-42 changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants