ARROW-243 Use libhdfs3

[rhl-]

This allows you to switch of libhdfs for libhdfs3. It seems to work locally. There are some issues with the existing HDFS IO Unit tests in that they don't necessarily properly configure themselves based on the configuration of the Hadoop cluster. (For example, disabling short circuit reads). It seems the previous PR hit some network hiccups. I'm trying to see if this retry makes travis ci pass.

[wesm]

You can git commit --amend (with no changes) to bump the commit hash and force push to trigger a Travis build, do not need to open a new PR. Moving my comment from #106 here:

Can you create a JIRA (on issues.apache.org) about this and add it (ARROW-XXX: ...) to the PR title?

I'm more in favor of adding a runtime (vs. compile/link-time) option to switch between libhdfs and libhdfs3. If we can avoid requiring any of these libraries at link-time that would be ideal. Note also that libhdfs3 features a transitive LGPL dependency (GNU SASL).

[rhl-]

I've gone ahead and created the JIRA issue.

I'm not sure what is wrong with the google test libraries on two of the builders. I'll have to take a closer look.

I'll look into making this change for libhdfs3.

Do you have any idea of the implementation of ReadAt is what is expected?

Do the travis builds test the hdfs code?