Truncate Support

[jakecharland]

Are there currently any plans to implement a WriteAt func? I think this would be a great addition to this already awesome library. It seems like this could be accomplished behind the scenes through the use of the truncate feature supported by hdfs.

[colinmarc]

Huh, interesting idea! I'll read about the truncate feature.

[colinmarc]

This pdf has some information on the semantics: https://issues.apache.org/jira/secure/attachment/12697141/HDFS_truncate.pdf

How were you imagining WriteAt working with that feature? To write into the middle of a file, we'd need to read all the rest of the file out somewhere, truncate, write the new bytes, and then write the rest of the file back. If you tried to WriteAt at offset 100 in a 10gb file, for instance, that would mean buffering up 10gb - 100 bytes, which seems untenable for a client.

Still, it'd be good to support truncate, so I'll leave this open for now. Neither of the tarballs we use for testing have support for it yet, so I'll wait for that. We'll also need to make sure it fails peacefully if talking to a namenode that doesn't support it, so that we have decent backwards compatibility.

[jakecharland]

Ya I think the WriteAt feature would not be attainable for the client with that scheme and I am not sure if there is really a way to avoid that (maybe a temp file). Truncate however would be really awesome to have.