Unexpected 412 Response from CloudBlockBlob.DownloadText()

[vinniep79]

Hello! I was hoping you could help clarify a behavior I'm seeing.

I'm using blob storage as a basic cache, and I may have multiple readers and a single writer. The blobs are just JSON representations of objects. Per the default strategy explained in Managing Concurrency in Microsoft Azure Storage:

If you do not explicitly specify a strategy last writes wins is the default.

Also, from Exam Ref (70-532) Developing Microsoft Azure Solution, Chapter 4:

Any updates made to a blob are atomic. While an update is in progress, requests to the blob URL will always return the previously committed version of the blob until the update is complete.

And that's exactly what I want. I realize that the storage library may have some different behavior, and that's what I wanted to clarify.

My basic cache read is doing the following:

        public CacheItem RetrieveItem(string container, string key)
        {
            CloudBlockBlob blockBlob = BlobHelper.GetBlob(container, key);
            blockBlob.FetchAttributes();

            CacheItem item = new CacheItem();
            item.Metadata = blockBlob.Metadata;
            item.Key = blockBlob.Name;
            item.Value = blockBlob.DownloadText();

            return item;
        }

My basic cache write is doing this:

        public void StoreItem(string container, CacheItem cacheItem)
        {
// parameter validation code omitted
            CloudBlockBlob blockBlob = BlobHelper.GetBlob(container, cacheItem.Key);

            // Add the value
            blockBlob.UploadText(cacheItem.Value);

            // Add metadata
            blockBlob.Metadata.Clear();
            foreach (var item in cacheItem.Metadata)
            {
                blockBlob.Metadata.Add(item);
            }

            // Commit metadata.
            blockBlob.SetMetadata();
        }

In production, we've occasionally gotten 412 responses from that .DownloadText() call. The stack trace looks like this:

Message: The remote server returned an error: (412) The condition specified using HTTP conditional header(s) is not met..

Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext):0
Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadToStream(Stream target, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext):29
Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.DownloadText(Encoding encoding, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext):17
XXXXXXXX.RetrieveItem(String container, String key):46

The particular blob that's causing this issue is 12 MB, so it seems entirely possible to me that I could have a write taking place (singleton writer) while there are reads happening.

Is this behavior expected? Is there any way I can get that "last write wins" / "requests return previously committed blob until the update is complete" behavior without getting 412 errors?

Thanks in advance!
Vinnie

[erezvani1529]

Hi @vinniep79,

This seems expected as the storage client locks on ETag during resume download to prevent data corruption (resume download happens after retry-able failures during download operations up to the specified number of retries in the RetryPolicy).

We can confirm this by checking server-side logs if you could send a timestamp/account info to ascl at microsoft dot com.

Thanks,
Elham

[vinniep79]

Hi @erezvani1529,

Thank you for the reply. So the behavior is expected. As to the 2nd part of the question: Is there any way I can get that "last write wins" / "requests return previously committed blob until the update is complete" behavior without getting 412 errors?

Even if I created one copy of the blob for reads, another for writes, and swapped them after an update, it seems like I'm just playing a race condition game.

[pemari-msft]

In this case, there are multiple reads being done by the client library. Each one is atomic, but there's no way to lock on to that version of the blob without 412s. The best way would be to do your own Download logic like the client library does, but to change how the recovery action works in the case of a partially committed blob (fail instead of retry, I presume).

This is the scenario:
In the case of downloading a blob that fails partway through, the client library has only two options:

1. try to finish downloading the blob from the position it failed, and if the blob was modified, then either fail with 412 or return a corrupt state
2. to fail and retry from the beginning, meaning that the blob would be in a state that did not exist at the time the read started.

You could write your own logic to do option 2 but the best one for the general customer case is option 1. Hope this helps!

[vinniep79]

@pemari-msft Thank you! Now I have the full picture.