support creating files bigger than 50GB

[kahing]

Right now each MPU part is fixed at 5MB, and since S3 has a 10000 maximum part limit we cannot create files bigger than 50GB. We can automatically adjust part size (ie: first 100 parts 5MB, then 50MB, etc) to support bigger files

[jindov]

is it support now?I still stuck at 10000 part with a 70GB file
ps: I've already git the latest version

[kahing]

Not yet. You can try manually increasing the part size from 5MB to something larger.

[jindov]

Thank you but goofys does not provide any option to increase part size. I must edit in source code?

[kahing]

I lift the limit to 100GB for now. The proper fix will come later.

[jindov]

Does we have any options to set part size manually yet?

[schelhorn]

Implementing larger part sizes (preferably automatically based on the file size) would be highly appreciated here as well; we are storing genomics files up to 500GB.

[kahing]

The difficulty is that when you write to a new file we don't know how large it's going to be. Will do some sort of staggering part sizes as mentioned in #20 (comment)

[kahing]

@jindov @schelhorn could either of you give the latest revision a try? You will need to have at least 125MB free memory since goofys buffers each part in memory.

this is not release quality just yet, I removed some gating code which means it is possible (although not likely) for goofys to flush thousands of parts concurrently, which is probably not what we want.

[kahing]

with the last fix I successfully wrote 900GB:

$ dd if=/dev/zero of=local-mnt/900GB bs=1M oflag=nocache count=900000
900000+0 records in
900000+0 records out
943718400000 bytes (944 GB) copied, 4834.32 s, 195 MB/s

test was done on a hi1.4xlarge which has 10GigE. goofys now supports writing up to 5MB * 1000 + 25 * 1000 + 125 * 8000 = 1005GB files.