Storage: Downloads give incorrect not found error if object overwritten between calls

[BrandonY]

Downloads from GCS currently seem to be accomplished as a two step process:

1. Fetch the current object's metadata, including a generation number.
2. Using the generation number returned in step 1, attempt to download the object.

Sample download showing an example of this:

client = storage.Client()
bucket = client.get_bucket('yarbrough-test')
blob = bucket.get_blob('test.txt')
blob.download_to_filename('/tmp/test.txt')

What this does:

GET /storage/v1/b/yarbrough-test?projection=noAcl
GET /storage/v1/b/yarbrough-test/o/test.txt
GET /download/storage/v1/b/yarbrough-test/o/test.txt?generation=1511999426608623&alt=media

The problem:

In the case where the object has been overwritten by a new generation between the metadata GET and the donwload GET, a "not found" error would be reported to the user even though at no point did an object with the specified name not exist. In other words, this happens:

Time 1: GET /storage/v1/b/bucket/o/object returns generation 12345.
Time 2: User in another thread creates a new generation of text.txt, generation 23456. Generation 12345 is deleted and now bucket "bucket" object "object" implies the new generation.
Time 3: GET /download/storage/v1/b/bucket/o/object?generation=12345 fails with a 404 because there is no longer a generation 12345.

Steps to reproduce:

1. Create a GCS bucket. Do NOT enable versioning.
2. Start a script which rapidly overwrites an object with a new generation.
3. Start a second script which repeatedly downloads the object.

You are likely to eventually run into a "404", although an object exists at all times.

How to fix:

I suggest this be fixed by retrying the download step if it returns with a 404 after the metadata is successfully fetched, unless the user explicitly asked for a particular generation number. Alternately, I might suggest skipping the metadata fetch step entirely, unless the user asks for some piece of metadata, which would significantly reduce the latency for small downloads.

[tseaver]

Alternately, I might suggest skipping the metadata fetch step entirely, unless the user asks for some piece of metadata, which would significantly reduce the latency for small downloads.

@BrandonY The Bucket.get_blob convenience wrapper is intended to fetch the metadata (which also ensures that the blob exists). To avoid that extra API call for a blob you know exists already, just construct the blob via Bucket.blob, e.g.:

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('yarbrough-test')
blob = bucket.blob('test.txt')
blob.download_to_filename('/tmp/test.txt')

Because the metadata is not loaded, this version also avoids tacking it onto the download URL, which means that the 404 will not occur either.

[tseaver]

I'm going to close this pre-emptively, as I think I've answered the question. Please feel free to reopen if I am mistaken.

[pstemari]

IT's very unclear to me how the get_blob call can be used safely. It seems to be impossible to get both the metadata and the content atomically. You can loop and retry until the download call succeeds, but that might never occur.

[BrandonY]

@tseaver: Thanks for the workaround. That's good to know.

That said, I think the get_blob() behavior is wrong. Using it can result in a NOT_FOUND error in cases where a resource of the input name is always available.

Retrying 2-3 times automatically to achieve consistency, or else failing completely with a CONFLICT or a custom error, would be significantly better behavior than failing with a NOT_FOUND error in a scenario where there is always a resource with the input name available. NOT_FOUND is surprising and leads the user on a wild goose chase looking for deleted objects.

[dhermes]

@BrandonY That's the nature of an eventually consistent system.

[pstemari]

GCS is a strongly consistent system, not just eventually consistent. Paul J. Ste. Marie | Bug Collector | pstemari@google.com | (206) 876-1968

…

On Mon, Jan 8, 2018 at 4:08 PM, Danny Hermes ***@***.***> wrote: @BrandonY <https://github.com/brandony> That's the nature of an eventually consistent system. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4499 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAzjxrFomTFSfK2hSoze8WFl3g-ZxSbUks5tIq4IgaJpZM4Qv0_B> .

[dhermes]

@pstemari IIUC that's only true of certain methods, e.g. objects.delete, but not true of others, e.g. objects.list

[BrandonY]

You are mistaken. GCS listing is strongly consistent. It was eventually consistent a while back, but we fixed that.

In addition, even if listing were eventually consistent, using generation numbers as a precondition allows you to detect that a conflict has occurred (and, in fact, get_blob does seem to take advantage of this fact. It just reports the failure as a 404 instead of recognizing that a conflict has occurred).

[dhermes]

That's great to hear, I'll stop spreading bad information immediately 😀