Create CustomExecutorsRequireFullPathRule to ease upgrade to Airflow 2.0

[turbaszek]

This issue is part of #8765

Rule

Create CustomExecutorsRequireFullPathRule which corresponds to

Custom executors is loaded using full import path

entry in UPDATING.md. This rule should allow users to check if their current configuration needs any adjusting
before migration to Airflow 2.0.

How to guide

To implement a new rule, create a class that inherits from airflow.upgrade.rules.base_rule.BaseRule.
It will be auto-registered and used by airflow upgrade-check command. The custom rule class has to have title,
description properties and should implement check method which returns a list of error messages in case of
incompatibility.

For example:

airflow/airflow/upgrade/rules/conn_type_is_not_nullable.py

Lines 25 to 42 in ea36166

	class ConnTypeIsNotNullableRule(BaseRule):
	title = "Connection.conn_type is not nullable"
	description = """\
	The `conn_type` column in the `connection` table must contain content. Previously, this rule was \
	enforced by application logic, but was not enforced by the database schema.
	If you made any modifications to the table directly, make sure you don't have null in the conn_type column.\
	"""
	@provide_session
	def check(self, session=None):
	invalid_connections = session.query(Connection).filter(Connection.conn_type.is_(None))
	return (
	'Connection<id={}", conn_id={}> have empty conn_type field.'.format(conn.id, conn.conn_id)
	for conn in invalid_connections
	)

Remember to open the PR against v1-10-test branch.

[pcandoalmeida]

Hi @turbaszek could I try working on this?

[potiuk]

I assigned you :)

[turbaszek]

@pcandoalmeida in case of any questions feel free to ask 👌

[dimberman]

Hi @pcandoalmeida have you made any progress on this ticket? If you do not have time I can take it over.

[pcandoalmeida]

Hi @dimberman yes by all means!

[dimberman]

@turbaszek @kaxil I'm a bit confused by this

https://github.com/apache/airflow/blob/master/airflow/executors/executor_loader.py#L81-L110 Looking at these lines it looks like Airflow 2.0 still allows loading from plugins? Does this need to be an upgrade check or deprecation warning?

[kaxil]

@turbaszek Is there a rule to do the same (use normal Python package / modules) for operators and sensors?

I couldn't find one but we removed the support for adding operators and sensors via Plugins so we should have a check for it.

[kaxil]

closed by #13678