Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ImportChangesRule for upgrade check #11056

Merged

Conversation

turbaszek
Copy link
Member

@turbaszek turbaszek commented Sep 21, 2020

Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.

Example:

Changes in import paths of hooks, operators, sensors and others
---------------------------------------------------------------
Many hooks, operators and other classes has been renamed and moved. Those changes were part of unifying names and imports paths as described in AIP-21.
The `contrib` folder has been replaced by providers directory and packages:
https://github.com/apache/airflow#backport-packages

Problems:
  1.  Using `airflow.contrib.sensors.python_sensor.PythonSensor` will be replaced by `airflow.sensors.python.PythonSensor`. Affected file: /files/dags/test_dag_1.py
  2.  Using `airflow.operators.bash_operator.BashOperator` will be replaced by `airflow.operators.bash.BashOperator`. Affected file: /files/dags/xyz.py
  3.  Using `airflow.operators.bash_operator.BashOperator` will be replaced by `airflow.operators.bash.BashOperator`. Affected file: /files/dags/generate_dags.py

Related to #8765


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@turbaszek turbaszek added the area:upgrade Facilitating migration to a newer version of Airflow label Sep 21, 2020
@turbaszek
Copy link
Member Author

cc @vikramkoka

@turbaszek

This comment has been minimized.

Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
@turbaszek turbaszek marked this pull request as ready for review September 24, 2020 14:02
@turbaszek
Copy link
Member Author

The docs faliure was already fixed.

@turbaszek
Copy link
Member Author

@olchas @mik-laj @TobKed can you please take a look?

Comment on lines +28 to +33
class ImportChange(
NamedTuple(
"ImportChange",
[("old_path", str), ("new_path", str), ("providers_package", Optional[None])],
)
):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be simplified

Suggested change
class ImportChange(
NamedTuple(
"ImportChange",
[("old_path", str), ("new_path", str), ("providers_package", Optional[None])],
)
):
class ImportChange(NamedTuple)
old_path: str
new_path: str
providers_package: Optional[None]

Copy link
Member

@kaxil kaxil Sep 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh wait, I know why you did this, for compatibility with Py 2.7 , Py 3.5 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, not a big fan of this but we this command has to work with 2.7

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some operators did the other way.. from providers to core
#10013
this message will be wrong for it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message will be:

Using airflow.operators.email_operator.EmailOperator will be replaced by airflow.operators.email.EmailOperator.

And I see no place for confusion here

@potiuk potiuk force-pushed the v1-10-test branch 2 times, most recently from bd4214b to 195f00d Compare October 3, 2020 22:42
@turbaszek turbaszek merged commit 4431c92 into apache:v1-10-test Oct 4, 2020
Airflow upgrade check automation moved this from In progress to Done Oct 4, 2020
@turbaszek turbaszek deleted the add-changed-import-rule branch October 4, 2020 11:48
RaviTezu pushed a commit to RaviTezu/airflow that referenced this pull request Oct 25, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
kaxil pushed a commit that referenced this pull request Nov 12, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
@potiuk potiuk added this to the Airflow 1.10.13 milestone Nov 14, 2020
@potiuk potiuk added the type:new-feature Changelog: New Features label Nov 14, 2020
potiuk pushed a commit that referenced this pull request Nov 16, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
kaxil pushed a commit that referenced this pull request Nov 18, 2020
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021
Adds new rule that will be run during airflow upgrade-check
and will look for old clasess and imports incompatible with
Airflow 2.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:upgrade Facilitating migration to a newer version of Airflow type:new-feature Changelog: New Features
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

5 participants