CLI to transfer between cloud storage and HDFS

[heuermh]

Was thinking that it might be useful to have a command line class in ADAM (or a transfer-only code path in existing commands) based on or adapted from Conductor that transfers between cloud storage and HDFS. Conductor is Apache 2 licensed but probably isn't written for use as a library. We could take advantage of our encrypted-HDFS-sensitive file copy mechanism for merging (e.g. -disable_fast_concat) and fix the upload issue for ADAM Parquet+Avro+metadata directories.

[fnothaft]

I'm not against this, but its hard to do this in a platform agnostic way, and it often requires you to depend on cloud vendor SDKs (which have lots of messy version compat issues and are often patched together in vendor distros). Also, the Hadoop FileSystem class allows many unimplemented methods, which makes things yet again worse (there's unfortunate backstory as to why I am mentioning this). My preference would be to have some sort of a pluggable way to plug in cloud-capable concatters, but that's also a bit of a mess.

[heuermh]

Closing as WontFix