New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when trying to upload multiple files (ASP.Net core) #580

Closed
Miojo opened this Issue Mar 9, 2017 · 15 comments

Comments

Projects
None yet
9 participants
@Miojo

Miojo commented Mar 9, 2017

Trying to upload multiple images to my S3 bucket. When i try to upload a single small image file (~10kB) the code works with no errors.... I am using the last version of the SDK

*Notes:
-No problems using Cloudberry to upload multiple files
-Sometimes the error does not occur, the upload is sucessfull. Then out of nowwhere it starts bugging again for some hours, the it works again for some time, then it bugs again for some time, and on and on...

The error is:

[Error] Error on StoreUploadedFiles
System.IO.IOException: The write operation failed, see inner exception. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally
   --- End of inner exception stack trace ---
   at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext()

The code is (something like):

foreach (var file in files){
    var request = new PutObjectRequest
    {
        BucketName = DefaultBucketName,
        Key = relativePath,
        InputStream = file.OpenReadStream(),
        AutoCloseStream = true,
        ContentType = file.ContentType,
        CannedACL = S3CannedACL.PublicRead
    };
    var result = await _amazonS3.PutObjectAsync(request);
}

@Miojo Miojo changed the title from Error when trying to upload multiple files to Error when trying to upload multiple files (ASPcore) Mar 9, 2017

@Miojo Miojo changed the title from Error when trying to upload multiple files (ASPcore) to Error when trying to upload multiple files (ASP.Net core) Mar 9, 2017

@PavelSafronov

This comment has been minimized.

Show comment
Hide comment
@PavelSafronov

PavelSafronov Mar 9, 2017

Contributor

Can you provide the full exception trace?
How large are the items when the error occurs?

bugging again for some hours

Do you mean that all uploads fail with this message for a few hours, then eventually start to work again?

Contributor

PavelSafronov commented Mar 9, 2017

Can you provide the full exception trace?
How large are the items when the error occurs?

bugging again for some hours

Do you mean that all uploads fail with this message for a few hours, then eventually start to work again?

@Miojo

This comment has been minimized.

Show comment
Hide comment
@Miojo

Miojo Mar 10, 2017

The files have an average size of 2mB. Sometimes I will try to upload 10+ at once.

Not ALL uploads fail, most of them fail (images of ~10KB will not fail, bigger than 1MB will fail). Then eventually it start to work again, (1-2hours after)

Cloudberry doesn't get affected.

here it is:

2017-03-09 18:55:03.290 +00:00 [Error] Error on StoreUploadedFiles
System.IO.IOException: The write operation failed, see inner exception. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally
   --- End of inner exception stack trace ---
   at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.RedirectHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MYPROJECT.Services.AWSFileSystemManager.<StoreUploadedFiles>d__4.MoveNext() in C:\Users\DEV1\Documents\GitHubPrivate\MYPROJECT_ASP\MYPROJECT\src\MYPROJECT\Services\FileSystemFacade.cs:line 80
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MYPROJECT.Services.PropertyFileManager.<StoreUploadedFiles>d__7.MoveNext() in C:\Users\DEV1\Documents\GitHubPrivate\MYPROJECT_ASP\MYPROJECT\src\MYPROJECT\Services\PropertyFileManager.cs:line 122
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

Miojo commented Mar 10, 2017

The files have an average size of 2mB. Sometimes I will try to upload 10+ at once.

Not ALL uploads fail, most of them fail (images of ~10KB will not fail, bigger than 1MB will fail). Then eventually it start to work again, (1-2hours after)

Cloudberry doesn't get affected.

here it is:

2017-03-09 18:55:03.290 +00:00 [Error] Error on StoreUploadedFiles
System.IO.IOException: The write operation failed, see inner exception. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally
   --- End of inner exception stack trace ---
   at Amazon.Runtime.HttpWebRequestMessage.<GetResponseAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.RedirectHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CredentialsRetriever.<InvokeAsync>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__10`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MYPROJECT.Services.AWSFileSystemManager.<StoreUploadedFiles>d__4.MoveNext() in C:\Users\DEV1\Documents\GitHubPrivate\MYPROJECT_ASP\MYPROJECT\src\MYPROJECT\Services\FileSystemFacade.cs:line 80
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MYPROJECT.Services.PropertyFileManager.<StoreUploadedFiles>d__7.MoveNext() in C:\Users\DEV1\Documents\GitHubPrivate\MYPROJECT_ASP\MYPROJECT\src\MYPROJECT\Services\PropertyFileManager.cs:line 122
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
@PavelSafronov

This comment has been minimized.

Show comment
Hide comment
@PavelSafronov

PavelSafronov Mar 10, 2017

Contributor

What OS are you running this on?
It seems that this may be an issue with Windows 10 - dotnet/corefx#4934

Contributor

PavelSafronov commented Mar 10, 2017

What OS are you running this on?
It seems that this may be an issue with Windows 10 - dotnet/corefx#4934

@Miojo

This comment has been minimized.

Show comment
Hide comment
@Miojo

Miojo Mar 10, 2017

The server is an EC2 instance:
AMI ID: Windows_Server-2016-English-Full-Base-2016.12.14 (ami-45e3ec52)

The files are being uploaded to S3

And I am browsing the website via Windows10 (probably doesn't matter anyways)

Miojo commented Mar 10, 2017

The server is an EC2 instance:
AMI ID: Windows_Server-2016-English-Full-Base-2016.12.14 (ami-45e3ec52)

The files are being uploaded to S3

And I am browsing the website via Windows10 (probably doesn't matter anyways)

@ferflores

This comment has been minimized.

Show comment
Hide comment
@ferflores

ferflores Aug 9, 2017

Did you find the problem? I am experiencing the same issue

ferflores commented Aug 9, 2017

Did you find the problem? I am experiencing the same issue

@guitarrapc

This comment has been minimized.

Show comment
Hide comment
@guitarrapc

guitarrapc Sep 20, 2017

I also experience the same issue on AWS EC2, Windows SErver 2012 Eng, with .NET Core 2.0 Console App.

guitarrapc commented Sep 20, 2017

I also experience the same issue on AWS EC2, Windows SErver 2012 Eng, with .NET Core 2.0 Console App.

@ferflores

This comment has been minimized.

Show comment
Hide comment
@ferflores

ferflores Sep 20, 2017

@guitarrapc running the .NET Core 2.0 Console App on linux this won't happen, I would like this to be fixed or get a workaround to get my apps working on windows but at least this does not happens on linux

ferflores commented Sep 20, 2017

@guitarrapc running the .NET Core 2.0 Console App on linux this won't happen, I would like this to be fixed or get a workaround to get my apps working on windows but at least this does not happens on linux

@guitarrapc

This comment has been minimized.

Show comment
Hide comment
@guitarrapc

guitarrapc Sep 21, 2017

Nice information ferflores! I'm testing several scenarios and found issue would not happen with Win10 Local desktop with a huge number of S3 operation but may happen on EC2.
Scenario summary :

  • instance type : i2.xlarge
  • over 300,000 file operation to S3 with .NET SDK async/await.
  • CPU would go to 100% for all cores. (4 cores)
  • Plenty of Memory free.
  • Network resource would be over 200MiB during Op.
  • Disk Read will be fully occupied by Op.

An exception seems happens when EC2's status check was failed, maybe S3 operation exhausted instance resource. Now I'm try throttling operation for both EC2 and S3. Let's say I believe the issue would not happen on Win2012 R2 w/.NET Core 2.0, stay tuned for final result;)

guitarrapc commented Sep 21, 2017

Nice information ferflores! I'm testing several scenarios and found issue would not happen with Win10 Local desktop with a huge number of S3 operation but may happen on EC2.
Scenario summary :

  • instance type : i2.xlarge
  • over 300,000 file operation to S3 with .NET SDK async/await.
  • CPU would go to 100% for all cores. (4 cores)
  • Plenty of Memory free.
  • Network resource would be over 200MiB during Op.
  • Disk Read will be fully occupied by Op.

An exception seems happens when EC2's status check was failed, maybe S3 operation exhausted instance resource. Now I'm try throttling operation for both EC2 and S3. Let's say I believe the issue would not happen on Win2012 R2 w/.NET Core 2.0, stay tuned for final result;)

@eduardobarbiero

This comment has been minimized.

Show comment
Hide comment
@eduardobarbiero

eduardobarbiero Oct 3, 2017

Here have same problem, with EC2, Windows Server 2012, using .NET Core 1.1 Interfaces API and running IIS 10.0.
@ferflores working fine on linux, atm?

eduardobarbiero commented Oct 3, 2017

Here have same problem, with EC2, Windows Server 2012, using .NET Core 1.1 Interfaces API and running IIS 10.0.
@ferflores working fine on linux, atm?

@ferflores

This comment has been minimized.

Show comment
Hide comment
@ferflores

ferflores Oct 3, 2017

@eduardobarbiero yes it is working fine on linux using the default linux ami of AWS

ferflores commented Oct 3, 2017

@eduardobarbiero yes it is working fine on linux using the default linux ami of AWS

@eduardobarbiero

This comment has been minimized.

Show comment
Hide comment
@eduardobarbiero

eduardobarbiero Oct 3, 2017

@ferflores I had success using Windows Server 2016/2012 and IIS 8.5 and 10.0.

I added the environment variables in IIS to "system.webServer/aspNetCore", "AWS_ACCESS_KEY_ID", "AWS_REGION" and "AWS_SECRET_ACCESS_KEY"

Thanks all!

eduardobarbiero commented Oct 3, 2017

@ferflores I had success using Windows Server 2016/2012 and IIS 8.5 and 10.0.

I added the environment variables in IIS to "system.webServer/aspNetCore", "AWS_ACCESS_KEY_ID", "AWS_REGION" and "AWS_SECRET_ACCESS_KEY"

Thanks all!

@sstevenkang

This comment has been minimized.

Show comment
Hide comment
@sstevenkang

sstevenkang Feb 2, 2018

Contributor

Seems like the root problem is in the Windows' WinHTTP. Since there's a work around and there's nothing SDK team can do to address this, I'll close it. Please open a new issue if there's something we can address in the SDK.

Contributor

sstevenkang commented Feb 2, 2018

Seems like the root problem is in the Windows' WinHTTP. Since there's a work around and there's nothing SDK team can do to address this, I'll close it. Please open a new issue if there's something we can address in the SDK.

@sstevenkang sstevenkang closed this Feb 2, 2018

@JonyTek

This comment has been minimized.

Show comment
Hide comment
@JonyTek

JonyTek Feb 6, 2018

Sorry to pull up an old issue but I am experiencing this issue. Can someone give me more info on the fix please?

I am seeing this on both my local Windows 10 env and ec2 instance with server 2016 DC. Dotnet core 2 web api.

JonyTek commented Feb 6, 2018

Sorry to pull up an old issue but I am experiencing this issue. Can someone give me more info on the fix please?

I am seeing this on both my local Windows 10 env and ec2 instance with server 2016 DC. Dotnet core 2 web api.

@AndreMessier

This comment has been minimized.

Show comment
Hide comment
@AndreMessier

AndreMessier Feb 9, 2018

@sstevenkang could you outline what the workaround is?

AndreMessier commented Feb 9, 2018

@sstevenkang could you outline what the workaround is?

@sstevenkang

This comment has been minimized.

Show comment
Hide comment
@sstevenkang

sstevenkang Feb 10, 2018

Contributor

From the comment thread above it seems like some customers had success patching or spinning up the latest windows 2016 instance.

I would be cautious of conflating the problem you are having with the original problem of this issue based on the specific excecption thrown.

Contributor

sstevenkang commented Feb 10, 2018

From the comment thread above it seems like some customers had success patching or spinning up the latest windows 2016 instance.

I would be cautious of conflating the problem you are having with the original problem of this issue based on the specific excecption thrown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment