load module cost too much time when parse dagbag in DagBag#process_file

[ZhendongBai]

the airflow scheduler load module cost too much time in method airflow.models.dagbag.DagBag#process_file
Airflow version: 1.10.4

What happened:

1. as we known , imp.load_source(mod_name, filepath) in airflow.models.dagbag.DagBag#process_file is a time cost operation,sometimes it may cost more than 1 minute, is where any better way to load module?
2. and I use the version 1.10.4, useimp.load_source(mod_name, filepath) to load module, and the version 2.x , use importlib to load module, so I want to ask a quastion - importlib is more efficent than imp ?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ashb]

No, importlib and imp use roughly the same code under the hood.

The "expensive" part is not the method used to load the module, but what ever your module is doing.

If it is taking 1 minute to load your DAG file, then I suggest moving imports or code execution out of the top level of the module.

[ZhendongBai]

@ashb thank you more than I can say, according my log , I make sure the dag py file import cost a lot of time, and they start to optimize the module import, thx very much , 😊 ~~