There is a new set of lines that cause big problems (https://github.com/airbnb/airflow/blob/master/airflow/executors/__init__.py#L11) --
# TODO Fix this emergency fix
try:
from airflow.contrib.executors.mesos_executor import MesosExecutor
except:
passI don't think anything from contrib can be loaded before airflow itself has finished loading, but MesosExecutor is loaded very early. So this chain of imports results:
from airflow.models import DAG (airflow/__init__.py)
--> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor (airflow/models.py)
--> from airflow.contrib.executors.mesos_executor import MesosExecutor (airflow/executors/__init__.py)
--> import airflow.contrib.hooks (airflow/contrib/__init__.py)
--> from airflow.hooks.base_hook import BaseHook (airflow/contrib/hooks/ftp_hook.py)
With the result that airflow.hooks is being imported well before it is supposed to be (we've barely even imported models), and this is causing all kinds of problems. Specifically, the ftp_hook isn't able to import a Connection because the import of models.py gets short-circuited. Normally, this would raise some kind of error but the MesosExecutor lines are suppressing it. The result is that airflow is missing chunks of its functionality at import time (though for some reason it becomes available later for certain objects... I don't understand Python internals well enough to guess why).
I think as a rule, an object in contrib can not be loaded as part of the Airflow startup. Contrib objects are like any user-created library -- designed to be instantiated after airflow is available. If MesosExecutor (or any other executors) are required as part of Airflow's internal setup (which seems to be the case), then they either need to live outside contrib or the entire contrib module needs to be rethought.
There is a new set of lines that cause big problems (https://github.com/airbnb/airflow/blob/master/airflow/executors/__init__.py#L11) --
I don't think anything from contrib can be loaded before airflow itself has finished loading, but MesosExecutor is loaded very early. So this chain of imports results:
With the result that airflow.hooks is being imported well before it is supposed to be (we've barely even imported models), and this is causing all kinds of problems. Specifically, the
ftp_hookisn't able to import aConnectionbecause the import ofmodels.pygets short-circuited. Normally, this would raise some kind of error but the MesosExecutor lines are suppressing it. The result is that airflow is missing chunks of its functionality at import time (though for some reason it becomes available later for certain objects... I don't understand Python internals well enough to guess why).I think as a rule, an object in
contribcan not be loaded as part of the Airflow startup. Contrib objects are like any user-created library -- designed to be instantiated after airflow is available. IfMesosExecutor(or any other executors) are required as part of Airflow's internal setup (which seems to be the case), then they either need to live outside contrib or the entire contrib module needs to be rethought.