[C++] Add "driver" option to HdfsClient to choose between libhdfs and libhdfs3 at runtime

[asfimport]

I would like to use for example libhdfs3 from pivotal to read apache parquet files. This would be a small change to the hdfs layer of apache arrow to support this.

Reporter: Ryan Lewis
Assignee: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-243. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
My preference would be to add an option to HdfsConnectionConfig to choose the client library used – to be loaded at runtime with dlopen – since libhdfs and libhdfs3 are designed to be roughly ABI-compatible. One reason for this is that binary distributions of Arrow may not know which libraries may be present on the target system.

This is more work than switching between the libraries at compile time but will be more flexible.

[asfimport]

Ryan Lewis:
Are you suggesting manually using dlopen? My guess would be that there is a way to achieve what you want without doing this. If the libraries really are ABI compatible, it should be possible to just swap out the underlying SO.

[asfimport]

Wes McKinney / @wesm:
Yes – part of the reason for using dlopen for libhdfs is that it's typically part of a Hadoop distribution, so libhdfs.so will likely not be in LD_LIBRARY_PATH. Since we already have this code in place (https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/libhdfs_shim.cc#L500), making libhdfs3.so also a soft (i.e. not loaded when libarrow_io.so is loaded) dependency isn't too much extra work.

This code would benefit from some refactoring (for example: there is only one possible set of function pointers available in the shim layer – you could put these in static shim structs) to make switching between the libraries as seamless as possible.

[asfimport]

Wes McKinney / @wesm:
[~me@ryanlewis.net] I did the refactoring we were talking about in #244. Give it a spin and let me know how it works for you.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 244
#244