Add Kerberos support to HDFS shim

[asfimport]

Speaking with @wesm, it would be really helpful to have Kerberos support in our HDFS logic. This should be straightforward; I would just need to switch to hdfsBuilderConnect() in the shim.

On a side note, is there a reason we aren't using Pivotal's libhdfs3? It uses RPCs natively rather than JNI.

https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3

Dask has Python wrappers for this.

https://github.com/dask/hdfs3

Reporter: Christopher Aycock / @chrisaycock
Assignee: Christopher Aycock / @chrisaycock

_{Note: This issue was originally created as ARROW-350. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
There is no reliable support for libhdfs3 – it was proposed as a patch to Hadoop and rejected. Meanwhile, a new official C++ HDFS client is in the works, see HDFS-8707. If there is a bug we cannot really count on the Apache Hadoop community to fix it.

Because libhdfs3 is mostly API compatible with libhdfs if someone will do the work we could add an option to switch between libhdfs and libhdfs3 in the HdfsClient factory function.

[asfimport]

Christopher Aycock / @chrisaycock:
PR is up:

#185

This builds on work for ARROW-345.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 185
#185