Invalid Syntax Error When Importing hdfs_hook

[amirulmenjeni]

Apache Airflow Version: 1.10.10

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration: N/A
• OS: CentOS Linux 7 (Core)
• Kernel: Linux myhost 3.10.0-1062.4.3.el7.x86_64 Improving the search functionality in the graph view #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: N/A
• Others: N/A

What happened:

Received syntax error when importing from airflow.hooks.hdfs_hook on Python 3.7.8. Here's the error message:

Traceback (most recent call last):
  File "subdags/copy_subset.py", line 9, in <module>
    from airflow.hooks.hdfs_hook import HDFSHook
  File "/home/amirul.menjeni/.pyenv/versions/dataops-flow-env/lib/python3.7/site-packages/airflow/hooks/hdfs_hook.py", line 24, in <module>
    from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
  File "/home/amirul.menjeni/.pyenv/versions/dataops-flow-env/lib/python3.7/site-packages/snakebite/client.py", line 1473
    baseTime = min(time * (1L << retries), cap);
                            ^
SyntaxError: invalid syntax

What you expect to happen:

No syntax error when importing.

How to reproduce it:

• Create python version 3.7.8 environment
• Install airflow
• Import airflow.hooks.hdfs_hook

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[amirulmenjeni]

Relevant: spotify/snakebite#216

[ephraimbuddy]

You need to install snakebite-py3, snakebite does not work with python 3. I'll try and address this in master

[amirulmenjeni]

Thanks, pip install snakebite-py3 fixed this issue.

[ephraimbuddy]

@mik-laj, snakebite-py3 is a fork of snakebite and is not maintained by Spotify. Do you suggest we should use it in place of snakebite?

[turbaszek]

Closing as the issue seems to be solved and it seems that snakebite-py3 is required for hdfs:

airflow/setup.py

Lines 296 to 298 in 12a652f

	hdfs = [
	'snakebite-py3',
	]