Fabric/Paramiko fails when uploading big files

[sebastien]

the file_write and file_upload method should support big (10M+) files. I had consistent failures with a 47Mb file.

The idea would be to split the file in chunks, and make sure that the chunks are transferred (using bz2 to compress the content, base64 to transfer as string and SHA256 as checksum).

[cybermnemosyne]

I was wondering why you don't just use scp ? I hit this error (along with another base64 alignment problem) and so switched to just using run scp followed by file_ensure

[sebastien]

Actually, that's a pretty good idea -- the only possible problems is that because scp will be run locally we'll have to resend the credentials (instead of using the already authenticated SSH channel used by Fabric).

[cybermnemosyne]

It seems to work ok and also has the benefit of not spitting out a dump of the file in the output - let me know if you want me to have a go at adding the method or changing it

[sebastien]

Hi David, it would be great if you could contribute something! For now I'd like to keep the current file_write as default until we know that the scp method works, so I would suggest that you write a file_write_scp(...) method and I'll add an option to use file_write_scp instead of file_write (it might be possible with variants).