Uploading large files seems super slow

[harshitha-bt]

I am trying to upload a ~500MB file to HDFS running on another node. This operation seems to take more than 10-15 minutes. Is this expected?

[colinmarc]

Hi @harshitha-bt,

This is probably due to #84, a really unfortunate performance regression. A fix is on the way.

[colinmarc]

(Should be fixed now, in cdda132 and v1.0.4)

[harshitha-bt]

My use case is to download the a multipart message of 700 MB from a rest endpoint and storing it locally and then did a client.CopyToRemote() which took 1 hour for ~700MB of data.
So, i tried using a fileWriter.Write(buf), but this is what i see from file_writer.go ( Write() ).

2017/08/27 20:25:38 offset and len of buffer = 0 706694149
2017/08/27 20:37:01 offset and len of buffer = 134217728 706694149

Even after the fix you mentioned here, it takes 10 mins to upload 1/7th of the file.
What is the best way to upload a file? CopyToRemote() or fileWriter.Write()?

[yetanotherbot]

Any update on this issue? I am seeing performance degradation from cross-zone uploads (also using fileWriter.Write(buf).