Skip to content

Support for streaming S3Path data #204

@omus

Description

@omus

After running into #197 I discovered that when using an S3Path as a I/O stream what is actually occurring is that a FileBuffer is being used which will load the entire S3 contents into an IOBuffer the first time any bytes are read.

AWS S3 supports the "Range" URI request parameter which allows request byte ranges. Effectively this allows use to perform a multi-part download of the file which should allow us to download chunks in parallel and also allow us to read a portion of the file without having to download all of the content. See the multi-part upload documentation for a list of advantages that should mostly also apply to downloading.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions