feat: server-mode streamed downloads #742
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Test Results 4 files ±0 4 suites ±0 4m 46s ⏱️ +6s Results for commit 96a590d. ± Comparison against base commit ce9b54a. This pull request removes 1 test.♻️ This comment has been updated with latest results. |
|
Code Coverage (Ubuntu)
Minimum allowed coverage is Generated by 🐒 cobertura-action against 96a590d |
|
Code Coverage (Windows)
Minimum allowed coverage is Generated by 🐒 cobertura-action against 96a590d |
sbrunato
force-pushed
the
handle-downloads
branch
2 times, most recently
from
4509ff8
to
a8f3e4f
Compare
sbrunato
force-pushed
the
handle-downloads
branch
13 times, most recently
from
59b3bd9
to
58576d0
Compare
sbrunato
force-pushed
the
handle-downloads
branch
5 times, most recently
from
acc445c
to
38ca15c
Compare
streams the zip files send by the provider directly to the user without storing them in the EODAG server container
for items where links to several asset files are given those assets can now be streamed directly without storing them or they are stored and a zip file is streamed
assets retrieved from a provider via the s3 rest plugin can now be streamed directly to the client or downloaded and streamed as a zip file
ensures that the download using the catalogs endpoint works as before and changes the order of the the file separators in the dowload stream
sbrunato
force-pushed
the
handle-downloads
branch
from
38ca15c
to
04288c0
Compare
sbrunato
force-pushed
the
handle-downloads
branch
from
04288c0
to
a2e42df
Compare
sbrunato
approved these changes
Jul 28, 2023
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #166
Description
With the changes in this pull requests the handling of downloads via the API is changed:
HTTPDownloadplugin zip files received from the provider (e.g.peps) will be directly streamed to the user without storing them. In case we receive individual asset files from the provider (e.g.earth_search_cog), they are also streamed and compressed on the fly streamed as a zip file.AwsDownload,S3RestDownloador external python APIs plugins, streaming is not implemented yet. As fallback, downloads are performed on the server and immediately deleted once transferred to the client.Here is the update download methods call graph (implemeted for
HTTPDownload):