Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: Avoid heavy top-level imports in operators
For complex DAGs, the import time added by this library could generate `DagBag` import timeouts if the configured value is low enough. An Airflow documented best practice is to [avoid heavy Python code](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code) that runs on DAG and Operator creation, and `dbt` imports are slow, based on profiling. Profiling can be easily run locally, with the following command: ```shell python -X importtime -c "from airflow_dbt_python.operators.dbt import DbtRunOperator" 2>import-times.log ``` And then parsed using a tool like [tuna](https://github.com/nschloe/tuna). Before this change, the operator import takes ~1.37s, which is reduced to ~0.25s with this fix. It's important to note that, from those 0.25s, more than 80% of the time is spent importing `airflow` components, which will be commonly already loaded in DAGs, so this library's import time for operators becomes insignificant.
- Loading branch information