Amazon.Runtime.AmazonServiceException often causing failures in getting documents from Amazon S3

[dyardyGIT]

We have a asp.net web application c#, .net framework 4.8 which gets documents/images from Amazon S3 and get the following exception at random times. I have yet to determine the pattern or the source of this issue yet?

This exception is thrown then, it starts working again but we are getting this randomly and not sure root cause. It is becoming a real quality issue for us.

We host in IIS on Windows Server 2012. Any ideas on where to start to resolve/eliminate this issue?

• [Info] - [An error occurred with the message 'A WebException with status Timeout was thrown.' when reading an object. Exception: Amazon.Runtime.AmazonServiceException: A WebException with status Timeout was thrown. ---> System.Net.WebException: The operation has timed out
  at System.Net.HttpWebRequest.GetResponse()
  at Amazon.Runtime.Internal.HttpRequest.GetResponse() in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler_bcl\HttpWebRequestFactory.cs:line 118
  at Amazon.Runtime.Internal.HttpHandler1.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 85 at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 41 at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:line 50 at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\Internal\AmazonS3ResponseHandler.cs:line 44 at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 72 --- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext executionContext, WebException exception) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\_bcl\WebExceptionHandler.cs:line 44 at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 204 at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 78 at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 46 at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 88 at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 46 at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 46 at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\Internal\AmazonS3ExceptionHandler.cs:line 48 at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\ErrorCallbackHandler.cs:line 44 at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:line 40 at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RuntimePipeline.cs:line 136 at Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller2 marshaller, ResponseUnmarshaller unmarshaller) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\AmazonServiceClient.cs:line 192
  at Amazon.S3.AmazonS3Client.GetObject(GetObjectRequest request) in E:\JenkinsWorkspaces\v3-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Generated_bcl45\AmazonS3Client.cs:line 2375
  at CH2MService.Core.Services.AmazonUtilityService.GetObject(String keyName) in e:\TFSBuild_work\WebApps_1_work\260\s\CH2MService\CH2MService.Core\Services\AmazonUtilityService.cs:line 68] - []

[klaytaybai]

Thanks for reporting this. Let us know if you find the problem. I'll have to think about it some. It would be easier to evaluate if it was always failing. You can search for timeout problems on the AWS Forums for S3 to see if anything helps you.

What are the timeout values that you are using? How large are your files?

[dyardyGIT]

The files are very small images.

I have tried using sdk getobject #1 and also
I have tried getting predefined url #2 for the amazonKey and using httpclient to pull the image.

I am suspecting something like it is failing within getobject call when the image/document is not available, and subsequent requests just fail.
I have also found in approach #2 that I am required to retry httpclient calls to get the image/document because the image/document is not available (but in this case it is httpclient call and i can put code around to retry).

In both cases, the image gets put on S3, then we get a list of document keys to immediately retrieve. I am starting to suspect that the image is actually not available immeidately or there is some lag from putting document there and having it available to immediately retrieve.
I started to think there may be something to this, when just yesterday I started #2 option (getting the url and using raw http to get the document) and had to use retry as the first request and/or second and third the image was not available to me immediately.

This might be the same reason getobject is failing too, however when it fails i think it is causing some failing exception to stick around longer causing us an issue. I have tried using retry around getobject but have not been been successful with retry code.

So is it possible that getobject is failing when image is not available. Again, one system is putting documents on S3 then giving the list of documents to another system to immediately retrieve. Is it possible there is some lag between a put and the document being available for a get?

[klaytaybai]

S3 does require eventual consistency, so that could certainly be a problem if you are trying to retrieve immediately after storing.

[klaytaybai]

If you wait for a little while, are you still getting these timeout exceptions?