Installation : Extra Packages

[afarbos]

Hi,

I see in your documentation some subpackage (http://pythonhosted.org/airflow/installation.html#extra-packages). But I never heard about that with pip, moreover I don't see them using pip search airflow.

[artwr]

Hi @glodus,
Packages extra are actually additional optional dependencies to enable optional functionality. Since airflow can connect with different databases, but we do not expect you to have all of them, you have the option of only installing the requirements you need.

You can see the detail here

[mistercrunch]

It's part of setuptools and supported by pip though subpackages don't seem to show when using pip search. You can dig for extras_require in here for more info: https://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies

For the full details of the packages and what they include, it's all in here:
https://github.com/airbnb/airflow/blob/master/setup.py#L56

The pip calls as documented should work, provided that you have the requirements for the specified packages to install. The reason why we created subpackages is that some packages are tricky to install and require some os level libs and we wanted to make it more modular where you just install what you need.

Note that the ipython project also uses extras_require

[afarbos]

Thanks for the answer, I all ready try the setup.py script but I have still so many warnings :

• WARNING - Couldn't import module postgres_hook
• WARNING - Couldn't import module hive_hooks
• WARNING - Couldn't import module presto_hook
• WARNING - Couldn't import module samba_hook
• WARNING - Couldn't import module hive_to_samba_operator
• WARNING - Couldn't import module hive_to_mysql
• WARNING - Couldn't import module mysql_to_hive
• WARNING - Couldn't import module hive_operator
• WARNING - Couldn't import module hive_stats_operator
• WARNING - Couldn't import module s3_to_hive_operator
• WARNING - Couldn't import module postgres_operator
• WARNING - Couldn't import module presto_check_operator

The setup.py should fix that no ? I try to do setup.py install.

Just like that, don't forget to add a shebang to setup.py (#!/usr/bin/env python2.7).

[afarbos]

I fix most of them pip -r requierements.txt.

If I have other question should I open new issues or do you have an ml or something like that?

[mistercrunch]

@glodus , the setup.py script doesn't install extras_require by default, I looked for switched before to try to make it work but couldn't find any. If you want all dependencies, you can do:
pip install -r requirements.txt

Some of the libs in there require os packages to be installed. You could also just follow the documentation and pip install airflow[subpackage]

[statwonk]

@mistercrunch @artwr I ran into this issue, too. I struggled until it occurred to me to look at the closed issues. I understand the reason you don't include the dependencies, yagni principle. That said, it might be good to find a way to suppress the warnings. As someone new to Airflow, the warnings were a huge distraction and I do believe they might dissuade others from going further.

[mistercrunch]

#28 addresses the warnings