Skip to content

Commit

Permalink
changed default download chunk size to 10MB
Browse files Browse the repository at this point in the history
issue528
this affects response.iter_content in the following functions:
- openeo.rest.connection.Connection.download
- openeo.rest.job.Job.download
- openeo.rest.userfile.UserFile.download

in Job.download the chunk size can still be overwritten by the user
  • Loading branch information
VictorVerhaert committed Feb 6, 2024
1 parent 7ce9292 commit 021ee46
Show file tree
Hide file tree
Showing 4 changed files with 5 additions and 4 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

### Changed

- Changed default `chunk_size` of various `download` functions from None to 10MB. This should improve download speed of large files.
### Removed

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion openeo/rest/connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -1566,7 +1566,7 @@ def download(

if outputfile is not None:
with Path(outputfile).open(mode="wb") as f:
for chunk in response.iter_content(chunk_size=None):
for chunk in response.iter_content(chunk_size=10000000):
f.write(chunk)
else:
return response.content
Expand Down
3 changes: 2 additions & 1 deletion openeo/rest/job.py
Original file line number Diff line number Diff line change
Expand Up @@ -351,13 +351,14 @@ def __repr__(self):
n=self.name, t=self.metadata.get("type", "unknown"), h=self.href
)

def download(self, target: Optional[Union[Path, str]] = None, chunk_size=None) -> Path:
def download(self, target: Optional[Union[Path, str]] = None, chunk_size=10000000) -> Path:
"""
Download asset to given location
:param target: download target path. Can be an existing folder
(in which case the filename advertised by backend will be used)
or full file name. By default, the working directory will be used.
:param chunk_size: size of chunks to download (in bytes). Default is 10MB
"""
target = Path(target or Path.cwd())
if target.is_dir():
Expand Down
2 changes: 1 addition & 1 deletion openeo/rest/userfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def download(self, target: Union[Path, str] = None) -> Path:
ensure_dir(target.parent)

with target.open(mode="wb") as f:
for chunk in response.iter_content(chunk_size=None):
for chunk in response.iter_content(chunk_size=10000000):
f.write(chunk)

return target
Expand Down

0 comments on commit 021ee46

Please sign in to comment.