Blob.rewrite() does not work with batches.

[bits01]

google-cloud-storage v1.8.0

Not sure whether rewrites are supposed to work when batched, but it would be nice and useful if they did, otherwise there's no efficient way to copy lots of blobs across buckets in different locations or with different encryption keys.

Example:

with gcs_client.batch():
    dest_blob.rewrite(src_blob)

Traceback:

Traceback (most recent call last):
  File "batch_test.py", line 10, in <module>
    dest_blob.rewrite(src_blob)
  File ".../pyvirtenv/python-common/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 1359, in rewrite
    rewritten = int(api_response['totalBytesRewritten'])
  File ".../pyvirtenv/python-common/lib/python2.7/site-packages/google/cloud/storage/batch.py", line 105, in __getitem__
    raise KeyError('Cannot get item %r from a future' % (key,))
KeyError: "Cannot get item 'totalBytesRewritten' from a future"

[tseaver]

I'm afraid that the current batch implementation does not deal well with any methods whose return values depend on the response payload (as opposed to just being able to update the relevant blobs / buckets from those responses when the batch completes).

Adding support for such methods would be a large undertaking.

[bits01]

Could the issue be left open and marked as a future enhancement request?

[chemelnucfin]

@bits01 I have added this issue to our feature request project

[bits01]

Thank you. Should the issue be re-opened then? It's currently marked closed.

[IlyaFaer]

I think, it can be done in fashion of: googleapis/google-cloud-python#8618. But first it's better to deal with 8618

[tseaver]

@IlyaFaer, what is the issue that needs to be dealt with first? googleapis/google-cloud-python#8616 doesn't seem related.

[IlyaFaer]

@tseaver, sorry, mistake, 8618. I'm waiting for review in it, and if changes will be accepted, we could use something alike in this issue

[verasativa]

Just to be clear: at this moment the most reliable way to copy a lots of files from one bucket to another (retry strategy, batch requests and object rewrite) in python is subprocess.run(["gsutil", "cp",...]) ?

I was comparing contributors here to gsutil repo, and looks like are totally different and isolated teams, why?

[tseaver]

Noted during review: Blob.rewrite returns a 3-tuple, (rewrite_token, bytes_rewritten, object_size), in order to permit tracking progress on the rewrite (it may only complete partially).

[billyvg]

This is also an issue when trying to use batch() to copy blobs while preserving the ACL of the original blob.

[cojenco]

Closing; We've added clarifications on the limited Batch support in the python client, see details in #1045