Upload big files directly to S3 #276
Conversation
There was a problem hiding this comment.
This looks good! Do we have to distinguish between compressed and uncompressed files?
The way the CLI works right now, FASTXPassthru will upload files however they are--compressed or uncompressed--and let the proxy uncompress them (if necessary) and recompress them. In contrast, FASTXInterleave will always uncompress, interleave, and upload the files uncompressed. The self._fsize attribute is actually the uncompressed size of both files summed together in FASTXInterleave and the file size on disk for FASTXPassthru. Both self._fsize are the actual file sizes that will be uploaded.
We probably don't have to guess that a 50GB compressed file will be uploaded as some larger amount: FASTXInterleave._fsize is the actual uncompressed file size already and FASTXPassthru._fsize is the file size that will be uploaded.
|
Ah you're right, we can probably have just one number then! |
No description provided.