-
Notifications
You must be signed in to change notification settings - Fork 35
Open
Labels
Description
After running into #197 I discovered that when using an S3Path as a I/O stream what is actually occurring is that a FileBuffer is being used which will load the entire S3 contents into an IOBuffer the first time any bytes are read.
AWS S3 supports the "Range" URI request parameter which allows request byte ranges. Effectively this allows use to perform a multi-part download of the file which should allow us to download chunks in parallel and also allow us to read a portion of the file without having to download all of the content. See the multi-part upload documentation for a list of advantages that should mostly also apply to downloading.