Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create ChainBetwenDAGAndOperatorNotAllowedRule to ease upgrade to Airflow 2.0 #11040

Closed
turbaszek opened this issue Sep 21, 2020 · 5 comments
Closed
Assignees
Labels
area:upgrade Facilitating migration to a newer version of Airflow good first issue kind:feature Feature Requests

Comments

@turbaszek
Copy link
Member

turbaszek commented Sep 21, 2020

This issue is part of #8765

Rule

Create ChainBetwenDAGAndOperatorNotAllowedRule which corresponds to

Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported

entry in UPDATING.md. This rule should allow users to check if their current configuration needs any adjusting
before migration to Airflow 2.0.

How to guide

To implement a new rule, create a class that inherits from airflow.upgrade.rules.base_rule.BaseRule.
It will be auto-registered and used by airflow upgrade-check command. The custom rule class has to have title,
description properties and should implement check method which returns a list of error messages in case of
incompatibility.

For example:

class ConnTypeIsNotNullableRule(BaseRule):
title = "Connection.conn_type is not nullable"
description = """\
The `conn_type` column in the `connection` table must contain content. Previously, this rule was \
enforced by application logic, but was not enforced by the database schema.
If you made any modifications to the table directly, make sure you don't have null in the conn_type column.\
"""
@provide_session
def check(self, session=None):
invalid_connections = session.query(Connection).filter(Connection.conn_type.is_(None))
return (
'Connection<id={}", conn_id={}> have empty conn_type field.'.format(conn.id, conn.conn_id)
for conn in invalid_connections
)

Remember to open the PR against v1-10-stable branch.

@turbaszek turbaszek added area:upgrade Facilitating migration to a newer version of Airflow kind:feature Feature Requests good first issue labels Sep 21, 2020
@turbaszek turbaszek added this to the Airflow 1.10.13 milestone Sep 21, 2020
@jmelot
Copy link
Contributor

jmelot commented Oct 2, 2020

Hello, could I please take this one?

@mik-laj
Copy link
Member

mik-laj commented Oct 2, 2020

@jmelot I assigned you to this ticket.

@jmelot
Copy link
Contributor

jmelot commented Oct 25, 2020

Thanks, and sorry for taking so long to follow up. I implemented a rule for this that parses the text of the files in the dags folder and looks for bitshift operators being used on DAG instances. I keep wondering if I'm missing a better way to do this, though. Would you please take a quick look and see if this is on track, or if there is a different way of detecting this I should be looking at instead?

@turbaszek
Copy link
Member Author

@jmelot would you mind opening a PR? This will simplify the review :) Please, target the v1-10-stable branch

jmelot added a commit to jmelot/airflow that referenced this issue Oct 25, 2020
turbaszek pushed a commit that referenced this issue Oct 27, 2020
…2.0 (#11839)

* Adding ChainBetwenDAGAndOperatorNotAllowedRule for checking upgrade to Airflow 2.0 (#11040)

* Cleaning up tests for #11839
@turbaszek
Copy link
Member Author

Closed via #11839

Airflow upgrade check automation moved this from To do to Done Oct 27, 2020
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this issue Mar 5, 2021
…2.0 (apache#11839)

* Adding ChainBetwenDAGAndOperatorNotAllowedRule for checking upgrade to Airflow 2.0 (apache#11040)

* Cleaning up tests for apache#11839
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:upgrade Facilitating migration to a newer version of Airflow good first issue kind:feature Feature Requests
Projects
No open projects
Development

No branches or pull requests

4 participants