gs: support progress callback

[mhham]

I may have missed an issue concerning this, but when I am dvc pushing to a google cloud remote, the progress bars that are currently displayed don't get updated progressively (They go from 0 when starting to 100 when finished).

It would be nice to have a dynamic progressbar with upload speed and remaining time estimation, especially when push huge sets of data.

[efiop]

Hi @mhham !

Indeed, callback for google cloud storate push/pull is not implemented yet. Let's keep this issue open as a reminder. Thank you for the feedback! 🙂

[casperdcl]

Problem is there's no current API.

google.cloud.storage.bucket.Bucket.:

• copy_blob
• upload_from_filename
• download_to_filename

Possibly could work around this by using upload_from_file and download_to_file with our own custom file objects which have progress for read() and write()

Looks good, let me look at it.

Let's focus on the downloading first.. .

with our own custom file objects

What do you mean @casperdcl ? Could you provide a link to that? I only found this
https://github.com/iterative/dvc/blob/38c210068ba5aeb70ad92a8c22e85832640d8ebd/dvc/api.py#L56
which doesn't seems to be helpful.

I think the more appropriate approach would be to use this google-resumable-media as it's recommended

For more fine-grained control over the download process, check out google-resumable-media. For example, this library allows downloading parts of a blob rather than the whole thing.

here.

I will check if using download_to_file with start & end gives the similar result to google-resumable-media. If so, we could avoid adding a new dependency.

.. with our own custom file objects which have progress for read() and write()

Ok, got it. I think @casperdcl meant the custom file object is to be written as a part of this task.

[casperdcl]

Based on the resumable media link it seems like the proposed download_to_file approach would be just as complicated to implement really. If you have issues though feel free to open an incomplete PR and I can take a look.

[casperdcl]

My main problem is testing - @efiop how would you recommended setting up local testing for google cloud storage push/pull?

[efiop]

@casperdcl I'm not aware of any local implementation that you could run on your machine 🙁 So far I was just using real GS to test things out.