Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create FernetEnabledRule to ease upgrade to Airflow 2.0 #11048

Closed
turbaszek opened this issue Sep 21, 2020 · 5 comments · Fixed by #11116
Closed

Create FernetEnabledRule to ease upgrade to Airflow 2.0 #11048

turbaszek opened this issue Sep 21, 2020 · 5 comments · Fixed by #11116
Assignees
Labels
area:upgrade Facilitating migration to a newer version of Airflow good first issue kind:feature Feature Requests

Comments

@turbaszek
Copy link
Member

This issue is part of #8765

Rule

Create FernetEnabledRule which corresponds to

Fernet is enabled by default

entry in UPDATING.md. This rule should allow users to check if their current configuration needs any adjusting
before migration to Airflow 2.0.

How to guide

To implement a new rule, create a class that inherits from airflow.upgrade.rules.base_rule.BaseRule.
It will be auto-registered and used by airflow upgrade-check command. The custom rule class has to have title,
description properties and should implement check method which returns a list of error messages in case of
incompatibility.

For example:

class ConnTypeIsNotNullableRule(BaseRule):
title = "Connection.conn_type is not nullable"
description = """\
The `conn_type` column in the `connection` table must contain content. Previously, this rule was \
enforced by application logic, but was not enforced by the database schema.
If you made any modifications to the table directly, make sure you don't have null in the conn_type column.\
"""
@provide_session
def check(self, session=None):
invalid_connections = session.query(Connection).filter(Connection.conn_type.is_(None))
return (
'Connection<id={}", conn_id={}> have empty conn_type field.'.format(conn.id, conn.conn_id)
for conn in invalid_connections
)

Remember to open the PR against v1-10-test branch.

@turbaszek turbaszek added area:upgrade Facilitating migration to a newer version of Airflow kind:feature Feature Requests good first issue labels Sep 21, 2020
@turbaszek turbaszek added this to the Airflow 1.10.13 milestone Sep 21, 2020
@Sangarshanan
Copy link
Contributor

Can pick this up : )

@turbaszek
Copy link
Member Author

@Sangarshanan I assigned you 👌

@turbaszek turbaszek moved this from To do to In progress in Airflow upgrade check Sep 22, 2020
@Sangarshanan
Copy link
Contributor

Heya : ) just wanted to confirm what needs to be done

Check if cryptography is not importable or if fernetkey is empty

Return a message saying that fernet will be enabled by default when upgrading and
an empty key has to be explicitly set in the fernet_key option of
section [core] in the airflow.cfg file to retain the current behaviour

Or am I going about this the wrong way?

@turbaszek
Copy link
Member Author

turbaszek commented Sep 23, 2020

Return a message saying that fernet will be enabled by default when upgrading and
an empty key has to be explicitly set in the fernet_key option of
section [core] in the airflow.cfg file to retain the current behaviour

as per https://github.com/apache/airflow/blob/master/UPDATING.md#fernet-is-enabled-by-default

So we should check if fernet_key key in config is empty. If yes we should inform users than when migrating they will have to change it explicitly to empty. If someone has something in this field - they are all good.

Also we should inform about this requirement:

At the same time, this means that the apache-airflow[crypto] extra-packages are always installed. However, this requires that your operating system has libffi-dev installed.

@turbaszek
Copy link
Member Author

Closed via #11116

Airflow upgrade check automation moved this from In progress to Done Sep 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:upgrade Facilitating migration to a newer version of Airflow good first issue kind:feature Feature Requests
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

3 participants