Chunked Encoding Issues

[mattjbrown]

I'm using Openstack.Net to upload CSVs from my web application. I've notice that sporadically I get this exception for a few minutes at a time when trying to upload:

[ProtocolViolationException]: Chunked encoding upload is not supported on the HTTP/1.0 protocol.
   at System.Net.HttpWebRequest.CheckProtocol(Boolean onRequestStream)
   at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at JSIStudios.SimpleRESTServices.Client.RestServiceBase.<>c__DisplayClass4.<Stream>b__3(HttpWebRequest req)
   at JSIStudios.SimpleRESTServices.Client.RestServiceBase.<>c__DisplayClass7.<ExecuteRequest>b__6()
   at JSIStudios.SimpleRESTServices.Client.RequestRetryLogic.Execute(Func`1 callback, IEnumerable`1 non200SuccessCodes, Int32 retryCount, Nullable`1 retryDelay)
   at JSIStudios.SimpleRESTServices.Client.RestServiceBase.ExecuteRequest(Uri url, HttpMethod method, Func`3 responseBuilderCallback, Dictionary`2 headers, Dictionary`2 queryStringParameters, RequestSettings settings, Func`2 executeCallback)
   at JSIStudios.SimpleRESTServices.Client.RestServiceBase.Stream(Uri url, HttpMethod method, Stream content, Int32 bufferSize, Int64 maxReadLength, Dictionary`2 headers, Dictionary`2 queryStringParameters, RequestSettings settings, Action`1 progressUpdated)
   at net.openstack.Providers.Rackspace.ProviderBase`1.StreamRESTRequest(CloudIdentity identity, Uri absoluteUri, HttpMethod method, Stream stream, Int32 chunkSize, Int64 maxReadLength, Dictionary`2 queryStringParameter, Dictionary`2 headers, Boolean isRetry, RequestSettings requestSettings, Action`1 progressUpdated)
   at net.openstack.Providers.Rackspace.CloudFilesProvider.CreateObject(String container, Stream stream, String objectName, String contentType, Int32 chunkSize, Dictionary`2 headers, String region, Action`1 progressUpdated, Boolean useInternalUrl, CloudIdentity identity)

Like I mentioned, its been happening pretty inconsistently, but it happens often enough that its preventing the use of the API bindings in otherwise production-ready code.

[sharwell]

@onesandzeros415 Is this something you've observed as well?

[onesandzeros415]

I've not been able to reproduce this.

[treefitty]

We see this from time to time too, so don't expect to be able to reproduce it immediately.... it's not a code logic issue, it appears to be some infrastructure event (or issue) that bubbles its way to the driver and causes this issue

We are working with Rackspace to resolve the problem as we have contacts internal... I will post back here if there is any resolution.

[treefitty]

Here is some more information (in the form of an email to Rackspace) about the issue and our resolution... our CTO wrote it:

----- start email -----

I was hasty in blaming something internal to our environment. It seems the problem is in the library supplied by you to connect to the cloud files (RackspaceOpenStack.Net). We're not sure why (perhaps a router restart or something) and it has put the component into an errored state.

While restarting the application resolved the problem it appears you have an issue you in your code. I suspect the settings on the Stream request are cached somewhere. In line 322 of RestServiceBase.cs, the value for settings.ChunkRequest was the problem and the settings are passed in to initiate the stream to the server. If this settings object was cached somewhere and for some reason the endpoint being streamed to was routed through a 1.0 proxy or something else on the network changed, you would find that the cached settings for the endpoint were no longer valid.

The solution was to restart IIS to force the app to restart and the settings.ChunkRequest to be reset to a valid value.

While this seems an obscure problem, the scale of Rackspace's infrastructure means somebody should definitely look at this for future versions of your cloud files integration library.

Please let me know if anybody needs more info on this.

[treefitty]

Yeeees, this just happened again, and as before restarting IIS resolved the problem

[nratcliff]

It's been happening to us as well, I just found this post while searching for a cause/resolution. It our case, IIS is not involved. I have a windows service that is responsible for moving files into the rackspace cloud. We'll get these errors in 5 - 15 minute bursts and then they go away. I assume this is an issue on the Rackspace side, but I'm not sure who to ask about it short of putting a ticket in and having them tell me they don't write the openstack client :-/

[sharwell]

I now have a reliable way to reproduce this issue!

[treefitty]

Can you share that little tidbit with us or do you reckon you can fix it?
Nice work either way :)

[sharwell]

I posted to Stack Overflow:
http://stackoverflow.com/q/22973178/138304

[treefitty]

Nice work mate and thanks for the update
Seems like an alright solution too... however I share your sentiment about wanting to understand the actual problem and how to fix it.

[sharwell]

More information...

This issue results from a conflict between the Microsoft .NET Framework (Mono is not affected) and the proxy configuration used for load balancing the Object Storage service. In particular, the implementation of the .NET Framework assumes that web services will follow RFC 2145 §2.3, and uses this assumption as justification for caching whether or not the service supports HTTP/1.1. In certain situations, the Object Storage service does not follow a "SHOULD NOT" clause in the RFC, which breaks the assumption. The solution for the SDK is to handle the ProtocolViolationException by "cleverly" (read: hacky, but not as hacky as reflection) resetting the ServicePoint.

[sharwell]

@mattjbrown @paulcarroll @nratcliff I can't send a pull request for resolving this issue until pull request #380 is merged. However, I would appreciate any commentary y'all have about the specific solution in 2d40127.

[nratcliff]

Nice work tracking this down and implementing a fix so quickly!

I read the response on SO and your solution makes sense based on 280Z28's description of ServicePoint's behavior.

There's only one potential issue I can see, but I don't think it's going to apply here. I learned the hard way that ServicePoint has an internal timer that fires every at an interval that is .5 * MaxIdleTime milliseconds. Any value lower than 2 causes the timer to fire continuously and the drives the CPU right to 100%. Seems unlikely to be an issue for the 1 millisecond you've got it set to 0. Just mentioning it in case someone down the road claims the latest version of the client spikes their CPU.

[sharwell]

@nratcliff The guard that checks the ProtocolVersion property restricts this code path to the very rare case it got set to HttpVersion.Version10. There is a 1ms window in this edge case where a call to Thread.Abort could prevent the MaxIdleTime from getting reset to the original value.

[nratcliff]

Completely agree. You'd have to have the ProtocolViolationException and a ProtocolVersion < 1.1. Like I said, it's never going to be an issue. My overly cautious side felt compelled to mention that side effect of the MaxIdleTime property.

[treefitty]

Really nice work guys, thankyou so much for tracking this down. I'll be sure to give the fixed branch a try!

[rebeccapowell]

I am also experiencing this issue whilst importing millions of files into Rackspace Cloud Files.

Is there an eta on the pull request? Or would I be better off forking your patch?

[treefitty]

Yes that's exactly the type of load that induces the issue for us too.
Personally I've forked the rackspace repo and added this fix myself.... and also applied the fix to ProviderBase~1 line 444 (ExecuteRESTRequest) as we saw the same exception come from here too even with the @sharwell fix.
I can't say absolutely that this has fixed all issues now as I've not got the new dll into production yet.

[rebeccapowell]

@paulcarroll Would you be interested to publish your fork publicly? I thought I had a fix but it hasn't worked. I still have another 8 million files still left to process! :-)

[treefitty]

yeah of course mate, not a problem, you can find it here: https://github.com/tempurer/openstack.net

[rebeccapowell]

@paulcarroll Thanks Paul. Looks good. I also reduced the 4 hour timeout to something more suitable for my purposes. I had noticed the application hanging for long periods of time. I think that was the cause.

[treefitty]

@benpowell yep, that's a really good call, I think I'll do the same... and introduce a retry X times on timeout
thanks

[rebeccapowell]

@paulcarroll your fork appears to be working successfully and these chunking errors fixed by @sharwell's fix have disappeared. This fix has really helped me. Thank you for all your help.