pyhive is installed without Hive dependencies

[snazzyfox]

Apache Airflow version: 1.10.10

(appears to also affect master)

What happened:

When airflow is installed with Hive support using apache-airflow[hive], using HiveServer2Hook to run a query throws the following exception:

...
  File "/usr/local/lib/python3.7/site-packages/airflow/hooks/hive_hooks.py", line 828, in get_conn
    database=schema or db.schema or 'default')
  File "/usr/local/lib/python3.7/site-packages/pyhive/hive.py", line 94, in connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pyhive/hive.py", line 152, in __init__
    import sasl
ModuleNotFoundError: No module named 'sasl'

What you expected to happen:

The error should not appear.

How to reproduce it:

Any minimal dag that uses HiveServer2Hook generates the error. Connection to a working Hive cluster is not required since the required dependencies are not installed

Probable Reason:

For pyhive to work with Hive, it should be installed as pyhive[hive]. The hive extra brings in the sasl package.

This is not caught testing since tests run with all the dependencies, and the packages required by pyhive[hive] happen to be also used for kerberos.

I know this will be a much bigger conversation, but maybe it's worth it to consider testing operators only with the dependencies they're supposed to rely on?

[ashb]

Wait, pyhive has an extra called hive....? 🤯

I know this will be a much bigger conversation, but maybe it's worth it to consider testing operators only with the dependencies they're supposed to rely on?

That would be amazing, yes, but would make our test suite take days to run to completion.

Still, might be something to think about.

[snazzyfox]

Yep, pyhive supports both hive and presto, so depending on the user's needs either pyhive[hive] or pyhive[presto] is installed.

Since airflow is just installing pyhive itself, it doesn't include the underlying libs for either. Maybe just change the dependency to pyhive[hive] for now works?

[eladkal]

Airflow doesn't need pyhive for Presto
#6822 removed that dependency. Presto is now using prestodb

[snazzyfox]

Yep, but pyhive is still needed for Hive. But since airflow is installing it without hive support, it's not actually usable unless packages for hive support are installed separately.

[potiuk]

Wait, pyhive has an extra called hive....?

That would be amazing, yes, but would make our test suite take days to run to completion.

Still, might be something to think about.

This is already planned for AIP-4 - system tests - combined with AIP-8 (likely) - split Airflow 2.0 into separate providers (follow up after AIP-21) and with AIP-26 Production image

That's one of the reasons why we refactored and moved everything to separate backport providers and have clear dependencies for all of them. This will allow to turn them into "airflow-providers" and once we setup automated system tests, we will be able to run those system tests separately for each provider - for hive for example by installing only hive provider (together with its dependencies) and running tests in a production image that will have only dependencies needed by Hive and run them in that image.

This is a mid-term goal I want to achieve (Still this year if possible)

BTW. @snazzyfox @eladkal - > do you think we need to add pyhive to the current set of dependencies for hive ? should we add it?

[snazzyfox]

Glad to hear that system tests will be coming! We should definitely still add it to the dependencies because that would actually fix it. System tests would only help catch the issue which we already know in this case :)