DELETE requests fail on Mono env

[ahmetb]

I am using client v3.0.3.0 on Mac OS X with Mono. All DELETE requests (delete container, CloudBlockBlob.Delete, CloudPageBlob.Delete) are failing with HTTP 403 Forbidden:

Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (403) Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.. ---> System.Exception: The remote server returned an error: (403) Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature..
at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.WebAsyncResult result) [0x0030c] in /private/tmp/source/bockbuild-mono-3.2.6/profiles/mono-mac-xamarin/build-root/mono-3.2.6/mcs/class/System/System.Net/HttpWebRequest.cs:1606

Seems like a signing issue. Repro is rather easy:

var account = new CloudStorageAccount (new Microsoft.WindowsAzure.Storage.Auth.StorageCredentials (STORAGE_ACCT, STORAGE_KEY), true);
var client = account.CreateCloudBlobClient ();
var container = client.GetContainerReference (Guid.NewGuid().ToString());
container.Delete (); // throws exception

It was also the same on v3.0.2.x, I just upgraded today to see if it's a known issue. Same version works fine on Windows. And please note, it's just DELETE operations, all others work fine.

Here's a dump of RequestEventArgs.Request.Header.ToString() caught in OperationContext.ResponseReceived event on both platforms,

OS X:

{User-Agent: WA-Storage/3.0.3 (.NET CLR 4.0.30319.17020; Unix 13.0.0.0)
x-ms-version: 2013-08-15
x-ms-client-request-id: b90b3f36-1f88-4150-a71c-20f1502a8e96
x-ms-date: Wed, 12 Feb 2014 06:27:20 GMT
Authorization: SharedKey f00f00f00:x+0U7Z9ggFl3MHuYfvrOR3wgieFLhQr/5j+sjaSxBnc=
Content-Length: 0
Connection: keep-alive
Host: f00f00f00.blob.core.windows.net

}

Windows:

{User-Agent: WA-Storage/3.0.3 (.NET CLR 4.0.30319.34003; Win32NT 6.2.9200.0)
x-ms-version: 2013-08-15
x-ms-client-request-id: e34b16cc-2921-4493-888c-307994d532d5
x-ms-date: Wed, 12 Feb 2014 06:29:47 GMT
Authorization: SharedKey f00f00f00:I9Gd72nFal6v8IPOYigOQsiJCYq90/VkDTLEsWGYWR8=
Host: f00f00f00.blob.core.windows.net
Connection: Keep-Alive

}

The only difference I see is Content-Length: 0 added on Windows probably while sending the request. I'll try to see if it is missing while signing. I'll be investigating a bit.

[ahmetb]

So I did some debugging on these 3:

1. Hmac256 implementation is broken on Mono.

2. Canonicalized string is computed differently on two platforms.

3. Request headers set is different.

1. Hmac256 implementation is broken on Mono.

This is not it. I copied CryptoUtility class over to mono and called it with the same inputs that Windows calls it and produced same output. Also, this signing issue is not happening intermittently, it's consistent.

2. Canonicalized string is computed differently on two platforms.

I added some log statements to SharedKeyAuthenticationHandler.SignRequest and used my private build for debugging on both Windows and Mono.

Canonicalized string computed same on both platforms (only random & time values differ):

Mono --> "DELETE\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:5fd5b700-c223-4e47-af09-76fb69b7c2ac\nx-ms-date:Wed, 12 Feb 2014 09:20:31 GMT\nx-ms-version:2013-08-15\n/f00f00f00/2d23ece0-519a-44a4-8e02-1cb9487e6f35\nrestype:container\ntimeout:90"
Win  --> "DELETE\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:af8961ee-c3a9-4436-8f27-0fbe33f4f039\nx-ms-date:Wed, 12 Feb 2014 08:44:46 GMT\nx-ms-version:2013-08-15\n/f00f00f00/cbfe97f5-7f57-4e38-9576-91a08fd3bffa\nrestype:container\ntimeout:90"

(I can add base64 values for those if needed.) But this also does not seem like the issue, so I thought, maybe modified headers are actually different than what's computed on canonicalization

3. Request headers set is different

But they also look the same. Here's what request.Headers.ToString() is on SignRequest():

On Mono:

User-Agent: WA-Storage/3.0.3 (.NET CLR 4.0.30319.17020; Unix 13.0.0.0)
x-ms-version: 2013-08-15
x-ms-client-request-id: 69dab96d-8a27-4538-aa02-c84e1e3ab635
x-ms-date: Wed, 12 Feb 2014 09:26:47 GMT
Authorization: SharedKey f00f00f00:AV0f+PVNYcrQx47u3RymTa3zpS6lCA8ciOla6bcLYzs=

On Windows

User-Agent: WA-Storage/3.0.3 (.NET CLR 4.0.30319.34003; Win32NT 6.2.9200.0)
x-ms-version: 2013-08-15
x-ms-client-request-id: 606ded4c-567c-4ec1-ba0f-3011787a28aa
x-ms-date: Wed, 12 Feb 2014 09:30:30 GMT
Authorization: SharedKey f00f00f00:UNWkBoMAjBk8+/Z0FnmPsIbSLM+Qs68fazMuueeMJwk=

(note these are different requests from the ones that I used in canonicalized string debugging.)

---

What else?

As I said, right now the only difference seemed like extra Content-Length: 0 header that appears on response.Request.Headers member of OperationContext.ResponseReceived event. (in fact my Delete() call in the repro has an operationContext parameter.)

So I am clueless at the moment, I probably need to install some mitmproxy on Mac & get Fiddler to capture storage https requests. I don't have experience in any, so any help is appreciated.

Hope all those helps.

[joeg]

Note, we have not explicitly tested the client library on mono.

That being said, one of the issues we have run into on some platforms is that the request wrapper object t(httpurlconnection / webrequest etc) inserts headers into the request without letting the client know. For example the HttpUrlConnection in Java will insert the Content-Length, but not allow the client to know what value it is sending. From the server perspective if it receives one of these headers that make up the canonicalized string it will attempt to honor them which in turn causes the authentication failure as the calculated signatures now differ. My guess is that you may be hitting something similar to this where the service is actually receiving Content-Length:0 even thought the client object does not expose this and it is not included in the string to sign. If you are debugging with the source and building a simple trick would be to edit the AppendCanonicalizedContentLengthHeader method (https://github.com/WindowsAzure/azure-storage-net/blob/master/Lib/Common/Core/Util/AuthenticationUtility.cs#L166) to append 0 in the case of deletes (under mono) and see if this fixes your issue. Also note, to assist in debugging, if you enable logging you can get the string to sign to help debugging these issues in the future.

[ahmetb]

I did enable the logging on operation context but I couldn't find where it does log. 😭 As far as I know, all modern HTTP/1.1 clients SHOULD send Content-Length header. (see content-length section here)

I don't know much about Fiddler, but when you send DELETE requests from Windows, does not .NET libraries automatically add Content-Length: 0? If it sends, that means there's nothing wrong with canonicalization.

Anyway, a fix may be needed :) Just saying. Thanks Joe.

[joeg]

Please see here for more information about enabling logging : http://blogs.msdn.com/b/windowsazurestorage/archive/2013/09/07/announcing-storage-client-library-2-1-rtm.aspx

Regarding Content-Length header, It is fine to send this header, however the client side abstraction must inform the application that it is doing so in order to authenticate properly. Regardless of platform it is fine to either send length=0 or not, so long as request.ContentLength reports the correct value to the client. (>-1L) so that it can be correctly signed. On windows this is not being sent automatically and hence it is not part of the canonicalized string, I believe what is happening in mono is that is sending this header, but when the client attampts to get the contentlength from the request it is not specified and therefore not part of the string to sign.

[ahmetb]

Hi Joe, I did a bit more of investigation. It indeed is caused by Content-Length: 0 header that is added by Mono runtime. Clearly it's not something that should be fixed in Storage Client. Also it doesn't report the header in request.Headers, for signing.

Here's a few mitmproxy on OS X dumps:

Fig 1. Mono sends that header:

Fig 2. Storage frontend uses that header (see that lonely 0 in the middle of canonicalized string):

Fig 3. Fiddler2 dump, .NET on Windows, Http clients don't send Content-Length header:

DELETE /79c1be5d-938e-4def-b10c-53f9538a393d?restype=container&timeout=90 HTTP/1.1
User-Agent: WA-Storage/3.0.3 (.NET CLR 4.0.30319.34003; Win32NT 6.2.9200.0)
x-ms-version: 2013-08-15
x-ms-client-request-id: b8188978-ea49-4af9-a697-848485a4b284
x-ms-date: Wed, 12 Feb 2014 23:27:58 GMT
Authorization: SharedKey f00f00f00:LJKqYSuIA0hg5Gq6DzDXU4XgpR/Tz8P7kMK9DiCeddY=
Host: f00f00f00.blob.core.windows.net
Connection: Keep-Alive

How should we resolve this? My opinion is, since having Content-Length header is preferable for HTTP/1.1, it can be added for DELETE requests, this way we can fix Mono and keep Windows stable.

[ahmetb]

Meanwhile I created Bug 17736 on Xamarin-Mono project.

[joeg]

Technically Mono should give back the correct header value for content length if it is indeed setting it to zero. An interesting experiment would be to attempt to set it to -1Lmanually and see if it still shows up in the request. I wouldn't want to force all clients to explicitly start sending the 0 length content length for deletes, however we could do a mono workaround where when we detect we are running on mono (see user agent string) we can automatically assume content length is sent for deletes which would unblock this scenario.

[ahmetb]

Setting Content-Length to -1, 0 or 1 (as strings) in SignRequest header throws

System.Exception("This header must be modified with the appropriate property.")

Anyway it's a Mono problem. Do you think headers['user-agent'].Contains('Unix') would suffice?

[ahmetb]

I was using request.Headers.Add, apparently for the default headers that's not supported. Anyway, adding this to the SignRequest method solves it:

if (request.Method == "DELETE")
    request.ContentLength = 0;

So adding this to all requests must have no harm except:

• It assumes all DELETE requests contain no request body
• Increase network communication by 20 bytes on Windows

Determining the Mono environment as a temporary workaround might be expensive if we use the recommended way: Type.GetType("Mono.Runtime") != null.

A cheaper way to determine if running on Mono could be assuming Environment.OSVersion.Platform == PlatformID.Unix is always Mono. That seems like a fair assumption to make:

if (Environment.OSVersion.Platform == PlatformID.Unix && request.Method == "DELETE")
{
    request.ContentLength = 0;
}

So which way should we go?

[ahmetb]

Fixed in Mono per request. https://bugzilla.xamarin.com/show_bug.cgi?id=17736 hoping to get and try soon.

[akorchemniy]

I ran into the same error when attempting to run:
table.CreateIfNotExists();

Turned out that my system time was off by a few hours.