Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

service/s3/s3manager: Should be able to track download progress #286

Closed
mwhooker opened this issue Jun 17, 2015 · 6 comments

Comments

@mwhooker
Copy link
Contributor

commented Jun 17, 2015

I want to show download progress to my users, which requires getting the total bytes to be downloaded.

I could do a HEAD request first to get this, but it's an additional round trip, and there's no telling if what I download is the same resource that I got the HEAD of.

Having access to downloader.written would also be useful.

If anyone can suggest a workaround, that would also be very helpful.

mwhooker added a commit to wercker/mhook that referenced this issue Jun 17, 2015

Move to official Amazon SDK
The official sdk gives us better support for resiliently downloading objects
from s3.

Unfortunately we lose the ability to show a progress bar with this move,
because we never get access to the request object. The trade off is that we get
parallel multipart downloads with the new S3Manager.

I've opened a ticket on the project to get access to some internals:
aws/aws-sdk-go#286

If support is added, I'll add back the ProgressBar. Until then, I don't think
it's worth hacking in.
@lsegal

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2015

I could do a HEAD request first to get this, but it's an additional round trip, and there's no telling if what I download is the same resource that I got the HEAD of.

This seems like a reasonable workaround to me. If you're worried about the object being concurrently written to while you are downloading it then the initial HEAD request will be the least of your concerns, as the downloader itself is likely making multiple concurrent GET requests-- and the object could theoretically change in between those requests as well.

But I agree that having progress support and totalBytes would be useful. The only problem is that they are not known at the start of the download, so if we exposed this it would likely be via function callback.

@mwhooker

This comment has been minimized.

Copy link
Contributor Author

commented Jun 17, 2015

If you're worried about the object being concurrently written to while you are downloading it then the initial HEAD request will be the least of your concerns, as the downloader itself is likely making multiple concurrent GET requests-- and the object could theoretically change in between those requests as well.

you're right. thanks for putting that into perspective.

mwhooker added a commit to wercker/mhook that referenced this issue Jun 17, 2015

Move to official Amazon SDK
The official sdk gives us support for concurrently downloading objects
from s3.

Unfortunately we lose the ability to show a progress bar with this move,
because we never get access to the request object. The trade off is that we get
parallel multipart downloads with the new S3Manager.

I've opened a ticket on the project to get access to some internals:
aws/aws-sdk-go#286

If support is added, I'll add back the ProgressBar. Until then, I don't think
it's worth hacking in.

mwhooker added a commit to wercker/mhook that referenced this issue Jun 17, 2015

Move to official Amazon SDK
The official sdk gives us support for concurrently downloading objects
from s3.

I've opened a ticket on the project to get access to some internals so that we
don't have to make a HEAD request to get object size:
aws/aws-sdk-go#286

If support is added, I'll remove the additional request.

mwhooker added a commit to wercker/mhook that referenced this issue Jun 17, 2015

Move to official Amazon SDK
The official sdk gives us support for concurrently downloading objects
from s3.

I've opened a ticket on the project to get access to some internals so that we
don't have to make a HEAD request to get object size:
aws/aws-sdk-go#286

If support is added, I'll remove the additional request.

@awsdave awsdave added blocked and removed blocked labels Nov 15, 2015

@xunchangguo

This comment has been minimized.

Copy link

commented Mar 14, 2016

Feature added?

@jasdel

This comment has been minimized.

Copy link
Member

commented Mar 14, 2016

Hi @xunchangguo this feature had not been added yet. It is still an outstanding item in our backlog. We're more than glad to discuss and review PR for feature requests for items such as this too :)

@jasdel jasdel changed the title expose downloader.totalBytes when using S3Manager service/s3/s3manager: Should be able to track download progress Apr 12, 2017

@petems

This comment has been minimized.

Copy link
Contributor

commented Feb 17, 2019

With a little help from @mwhooker, I made an example of how to do this for an S3 download.

Inspired by the PR to show this for upload, I added an example PR here: #2456

@diehlaws

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2019

Example added in 07f5610 - closing issue.

@diehlaws diehlaws closed this Jun 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.