CLI to transfer between cloud storage and HDFS #1896
Closed
Milestone
Comments
I'm not against this, but its hard to do this in a platform agnostic way, and it often requires you to depend on cloud vendor SDKs (which have lots of messy version compat issues and are often patched together in vendor distros). Also, the Hadoop |
Closing as WontFix |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Was thinking that it might be useful to have a command line class in ADAM (or a transfer-only code path in existing commands) based on or adapted from Conductor that transfers between cloud storage and HDFS. Conductor is Apache 2 licensed but probably isn't written for use as a library. We could take advantage of our encrypted-HDFS-sensitive file copy mechanism for merging (e.g.
-disable_fast_concat
) and fix the upload issue for ADAM Parquet+Avro+metadata directories.The text was updated successfully, but these errors were encountered: