'Blob.exists()' does not work within batch context

[ludwigschubert]

The Blob.exists() method does not work when run within a Batch context. The normal behavior of exists() is to return True unless a NotFound exception occurs. Within the Batch context the exception seems to be suppressed and the function returns True. After leaving the Batch context, an Exception is then thrown.

This is how I expected to be able to use the exists() function:

blobs = [storage.blob.Blob(path, bucket) for path in paths]
with client.batch():
  bools = [blob.exists() for blob in blobs]

Without the Batch contextmanager this code works, if inefficiently. With the Batch contextmanager the code returns all Trues and throws an exception when leaving the context.

This behavior seems unintuitive to me. Please let me know if the API is meant to be used differently. If it is meant to be used as in the provided code sample, I'd be happy to attempt a fix if one of the maintainers could point me in the right direction.

Environment configuration just in case:

• macOS 10.13.3
• Python 3.6.5
• google-cloud-storage==1.8.0

[tseaver]

@ludwigschubert Under the covers, Blob.exists() makes a GET request to the blob's resource URL and converts a 404 response into a False, 20x into a True. This implementation does not fit well with the current Batch design, which fakes the responses inside the context manager, and then applies the sub-responses to the individual targets at exit. See the comment in Blob.exists.

[davidbernat]

To be clear: when this was moved to "feature request" by @frankyn , the implied statement is:

there is no way to use blob.exists() within a batch context

Is this correct?

Thanks!

[davidbernat]

For @ludwigschubert and those who arrive via Google.

I found a simple fix for this: wrap each call for blob.exists() in a try/catch.

def to_parallelize_catch_exceptions(f, on_error_value, *args):
    try:
        return f(*args)
    except:
        return on_error_value

def parallelize(blobs, function, args_lists=None, on_error_value=None, n_threads=50):
    if args_lists is None:
        args = [(function, on_error_value, blob) for i, blobs in enumerate(stubs)]
   else:
        args = [(function, on_error_value, blob, *args_lists[i]) for i, stub in enumerate(blobs)]
    with multiprocessing.Pool(min(n_threads, len(blobs))) as p:
       values = p.starmap(to_parallelize_catch_exceptions, args)
   return values

def blob_exists(blob):
    blob.exists()

blobs = [....]
with client.batch():
    parallelize(blobs, blob_exists, on_error_value=False)

Of course, there are two layers of abstraction above which are not expressly necessary, but I've copy-pasted my internal code that handles the parallelization for all the cloud client functions. Feel free to comment or DM me for more information as necessary. Cheers.

[tseaver]

Possible implementation at googleapis/google-cloud-python#8618

[cojenco]

Thank you folks for providing above workarounds!

The current batch design does not support library methods whose return values depend on the response payload. In this case, the error handling and conversion in blob.exists() is not fully supported in a Batch context.

However, note that a new raise_exception flag is added to Batch via #1043 (pending release in v2.10.0). Setting raise_exception=False allows all exceptions to be included in list of return responses. Although not recommended, it is now possible to get a list of 404 responses calling blob.exists() with storage_client.batch(raise_exception=False)

In addition, we've also added clarifications on the limited Batch support in the python client, see details in #1045