Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
CLI to transfer between cloud storage and HDFS #1896
Was thinking that it might be useful to have a command line class in ADAM (or a transfer-only code path in existing commands) based on or adapted from Conductor that transfers between cloud storage and HDFS. Conductor is Apache 2 licensed but probably isn't written for use as a library. We could take advantage of our encrypted-HDFS-sensitive file copy mechanism for merging (e.g.
I'm not against this, but its hard to do this in a platform agnostic way, and it often requires you to depend on cloud vendor SDKs (which have lots of messy version compat issues and are often patched together in vendor distros). Also, the Hadoop