RFC: add a put method on Client that mimic put command in hdfs dfs cli tool

[fMeow]

Put files to hdfs in hadoop cli client

When uploading a file, the hadoop hdfs shell client first create a tmp file with a ._COPY_ suffix, and later rename this file. This way we can prevent reading an incomplete file.

The following command will first create a file /hdfs/path/local-file.log._COPY_ on the hdfs and later rename it to /hdfs/path/local-file.log. In this way we always get a completely uploaded file if we ignore files that end with ._COPY_.

$HADOOP_HOME/bin/hdfs dfs -fs hdfs://default/ -put ./local-file.log /hdfs/path/

What about hdrs

hdrs can surely mimic this behaviour and I think it profitable to add an method to Client to implement this behaviour.

Here is a quick MVP:

impl Client {
    pub fn put_file<T: AsRef<[u8]>>(&self, path: &str, content: T) -> io::Result<()> {
        // TODO maybe aditional check that the file does not exist
        // TODO check that the path is not a directory

        let tmp = format!("{}._COPY_", path);
        let mut file = self.open_file().append(false).write(true).open(&tmp)?;
        file.write_all(content.as_ref())?;
        self.rename_file(&tmp, path)?;

        Ok(())
    }
}

[Xuanwo]

Thanks for your RFC first!

Adding support for rename_file LGTM.

But hdrs intends to be the hdfs lib that used by hdfs dfs, which means pub_file is not suitable to add. I think a rust port of hdfs dfs is good place for put_file.

[Xuanwo]

Thanks for the RFC again, hdrs will not implement this feature.