Create TaskHandlersMovedRule to ease upgrade to Airflow 2.0

[turbaszek]

This issue is part of #8765

Rule

Create TaskHandlersMovedRule which corresponds to

GCSTaskHandler has been moved, WasbTaskHandler has been moved, StackdriverTaskHandler has been moved , S3TaskHandler has been moved, ElasticsearchTaskHandler has been moved, CloudwatchTaskHandler has been moved

entry in UPDATING.md. This rule should allow users to check if their current configuration needs any adjusting
before migration to Airflow 2.0.

How to guide

To implement a new rule, create a class that inherits from airflow.upgrade.rules.base_rule.BaseRule.
It will be auto-registered and used by airflow upgrade-check command. The custom rule class has to have title,
description properties and should implement check method which returns a list of error messages in case of
incompatibility.

For example:

airflow/airflow/upgrade/rules/conn_type_is_not_nullable.py

Lines 25 to 42 in ea36166

	class ConnTypeIsNotNullableRule(BaseRule):
	title = "Connection.conn_type is not nullable"
	description = """\
	The `conn_type` column in the `connection` table must contain content. Previously, this rule was \
	enforced by application logic, but was not enforced by the database schema.
	If you made any modifications to the table directly, make sure you don't have null in the conn_type column.\
	"""
	@provide_session
	def check(self, session=None):
	invalid_connections = session.query(Connection).filter(Connection.conn_type.is_(None))
	return (
	'Connection<id={}", conn_id={}> have empty conn_type field.'.format(conn.id, conn.conn_id)
	for conn in invalid_connections
	)

Remember to open the PR against v1-10-test branch.

[ephraimbuddy]

I'm interested in this issue

[ephraimbuddy]

But it seems like this could be added to the import changes rule PR #11056?

[turbaszek]

@ephraimbuddy this is more like about checking that users configuration in airflow.cfg uses old task handlers in task_log_reader. Similar to #11067

[ephraimbuddy]

Got it

[ephraimbuddy]

The remote logging documentation for the stable release is the same as the master documentation.

2.0 doc: https://airflow.readthedocs.io/en/latest/logging-monitoring/logging-tasks.html
stable doc: https://airflow.readthedocs.io/en/stable/howto/write-logs.html

I'm finding it difficult looking for what to check in previous version that has changed in 2.0.
What I have found so far is the change in import path as can be seen here :
https://github.com/apache/airflow/blob/master/UPDATING.md#airflowcontributilslog-has-been-moved.

What do you think?

[turbaszek]

What I have found so far is the change in import path as can be seen here :
https://github.com/apache/airflow/blob/master/UPDATING.md#airflowcontributilslog-has-been-moved.

That's exactly the change. I see that the UPDATING.md has changed a bit since we created the list of rules... We should check if users configuration uses one of those handlers and warn them that this will change in 2.0

[ephraimbuddy]

I'm suggesting that we add it in import change PR #11056 just like we added it in the deprecated classes here:

airflow/tests/deprecated_classes.py

Line 1783 in 51181e8

ALL = HOOKS + OPERATORS + SECRETS + SENSORS + TRANSFERS + UTILS + LOGS

,

This will help us not to duplicate code. What do you think?

airflow/tests/deprecated_classes.py

Lines 1756 to 1781 in 51181e8

	LOGS = [
	(
	"airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
	"airflow.utils.log.s3_task_handler.S3TaskHandler"
	),
	(
	'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler',
	'airflow.utils.log.cloudwatch_task_handler.CloudwatchTaskHandler'
	),
	(
	'airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler',
	'airflow.utils.log.es_task_handler.ElasticsearchTaskHandler'
	),
	(
	"airflow.providers.google.cloud.log.stackdriver_task_handler.StackdriverTaskHandler",
	"airflow.utils.log.stackdriver_task_handler.StackdriverTaskHandler"
	),
	(
	"airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
	"airflow.utils.log.gcs_task_handler.GCSTaskHandler"
	),
	(
	"airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
	"airflow.utils.log.wasb_task_handler.WasbTaskHandler"
	)
	]

[turbaszek]

The difference is that the #11056 is for DAGs, task handlers are used in airflow.cfg that is not validated in #11056 So we need to look for problems in two different places

[ephraimbuddy]

Ok. It's clear now

[turbaszek]

Closed via #11265