GetObjectRequest hangs randomly. #152
Comments
|
P.S. we are currently running AWS 2.3.11 but the issue was present with 2.3.9 also. |
|
http://support.microsoft.com/kb/980817 perhaps is a likely culprate? Only thing is im pretty sure were running .net 4.5 |
|
Using the following code works, so i believe there's an issue with the sync version in the SDK: using (var amazonS3Client = CreateS3Client())
{
var s3PartDownloadRequest = new GetObjectRequest
{
BucketName = download.Bucket,
Key = download.ObjectKey,
ByteRange = new ByteRange(position, end)
};
using (var getObjectResponse = amazonS3Client.GetObjectAsync(s3PartDownloadRequest).GetResult())//async with my own sync method extension.
{
getObjectResponse.WriteResponseStreamToFile(dest);
}
}
//GetResult:
public static T GetResult<T>(this Task<T> task)
{
task.ConfigureAwait(false);
task.Wait();
return task.Result;
} |
|
Bug also observed with "UploadPart" calls(non async version). i belive there is a bug in your "Invoke" flow. |
|
Thanks @Stormsys for reporting this issue and providing details! I'll look into this. I have a few questions about your application.
Because the hang is intermittent, it would be helpful to replicate this issue under different conditions.
|
|
Hi, Just to add, with upload part even with my sync wrapper the bug persists, even Async().wait() never detects a completion state sometimes, interestingly we have not seen this in GetObject since we switched over to the async version. To awsner your questions:
and i can certainly try http, but realistically this probably would not be an satisfactory workaround. |
|
Thanks @Stormsys , yes I understand, I suggested trying out http to isolate the behavior. |
|
@gokarnm i will try the Http setting tomorrow, we do have another server we can test on but have not yet done so, can also do this if it helps. |
|
Has anyone been able to resolve this issue? I'm facing it too... |
|
I also experience this sometimes, UploadPartAsync never returns, and actually, it doesn't even upload, since the StreamTransferProgress handler doesn't get anything. I also provide a CancellationToken, and when the problem occurs, it even ignores the cancellation, so there's actually no way to get this task finished. |
|
In case anyone still has this issue, I've found two changes that removed the problem:
Two more comments:
|
|
I am seeing something very similar, although we are using the PutObject method. Running "netstat -a -n | find /c "54.231." indicates the same number of connections These connections remain in this state until the application is terminated (this has caused problems when using the library in a website......it must be restarted to clear). My guess is that the underlying httpwebrequest connection is not being closed properly. We have tried (without success) setting... |
|
I think the key might be to dispose of the GetObjectResponse as quickly as possible. In my program, I'm concurrently downloading the contents of an entire bucket (with 35 concurrent worker tasks). I noticed that I started seeing object requests hang indefinitely if I didn't immediately read the contents of GetObjectResponse.ResponseStream into memory and dispose of the stream. |
|
I'm using the Quartz Scheduler for .Net and I was receiving the problem intermittently during a Job Execution. I have implemented Kobi's and Stormsys's posts and they both improved matters - but didn't fully fix the problem. The only thing that HAS managed to fix the problem has been to make a new instance of my S3Client in my Job Execution class whenever it executes rather than resolving it using dependency injection - I think it has something to do with thread safety. After implementing this, along with Kobi's and Stormsys's posts I haven't experienced the problem any more. |
|
I have a related problem, using the asp.net core SDK version Does the AmazonS3Client have any synchronization or shared state? |
|
I've been experiencing this, and on investigating the issue I think the line to blame is https://github.com/aws/aws-sdk-net/blob/aws-sdk-net-v2/AWSSDK_DotNet35/Amazon.Runtime/Pipeline/HttpHandler/HttpHandler.cs#L104. |
|
The new 3.3.1 version of Core contains the PR #449 which addresses this problem. Please let us know if the problem persists. Thanks! |
This is the latest release version and contains a fix to aws/aws-sdk-net#152 Which is likely the case of our "hangs when reading from queues" issue We have a workaround, but the internal fix is IMHO better
This is the latest release version and contains a fix to aws/aws-sdk-net#152 Which is likely the case of our "hangs when reading from queues" issue We have a workaround, but the internal fix is IMHO better. The question is, do we still need the "linkedCancellationToken" measure that fixes it? If we view it as a fix to a very specific bug that we are sure is fixed then no. If we view it as a general "belt and braces" reliability measure then yes.
|
Closing due to inactivity. If any of you guys encounter this issue again, feel free to reopen it. Thanks! |
|
I'm not sure how to reopen this issue. So adding my problem here. We have a server side application in .Net developed using Nancyfx framework. Running as selfhost. This application receives multipart request with multiple files(around 100). All these files are supposed to be upload to S3 bucket. Using Parallels to upload files to S3 bucket. Right now creating s3 object for every task. This is causing too many HTTP connections and after a while system is become slow or s3 latency increases. We have optimized at TCP level, to reduce the TcpWaitTimeDelay to 30 seconds. Can anyone help how to resolve this issue ? Using Windows 2012 Let me know if any further information require. Thanks in advance. |
|
Separate issue was opened for this question, no need to re-open - #546 |
|
Hangs on AWSSDK.Core 3.3.12 + AWSSDK.S3 3.3.5.12. |
|
From our experience concurreny doesn't really matter, if you don't dispose, the connection will hang around until the application terminates. We are simply disposing of the object as soon as possible and all the problems have disappeared. |
|
@randall-peakey-com: Interesting. Okay, this will be my next attempt at a fix. Right now we return the whole GetObjectResponse and handle it elsewhere in the code, so I might just have to rewrite it to use only the stream and copy it out to a MemoryStream and return that. Thanks for the info. |
|
Alternatively, just rewrite everything that was using the GetObjectResponse to just be inside a using statement and it's actually done the trick, even without the DefaultConnectionLimit change. Thanks @randall-peakey-com! |
|
We have been hitting this issue sporadically in our application for over a year - always while hitting the S3 api concurrently on multiple threads. We just hit again tonight and I got a stack trace of the hung thread. See below. We have hit this on different sdk calls: GetObject, GetObjectMetadata, and PutObject. Our situation was greatly improved a while ago by auditing every call to the sdk to make sure we weren't leaking requests or s3 clients. Here's the stack trace I captured tonight. At the time we detected this hang, we had 16 threads concurrently hitting s3. This on on version 3.3.0 of the sdk on .Net Framework 4.7, Windows Server 2016, running on a machine inside EC2. Does anything jump out here? I haven't tried moving all our S3 access to http (as ooposed to https) as I see suggested above. Is that still a recommended workaround? |
|
This may be related to https://github.com/dotnet/corefx/issues/21796 (.NET FX deadlock on sockets cleanup and due to ServicePoint.set_ConnectionLimit internal locks) I've started getting deadlocks due to many S3 SDK GetObject/Metadata/Exists calls and eventually something deadlocks exactly like the above issue. |
|
I've posted a workaround that is currently working for us, I believe: https://github.com/dotnet/corefx/issues/21796#issuecomment-381515493 |
diehlaws
added
guidance
|
@djluck commented on Mar 21, 2016:
This was exactly my problem. We were mistakenly calling |
|
It's April and 2019 and, years later, we are STILL experiencing random and sporadic hangs when accessing S3 via the .Net AWSSDK with a high degree of parallelism. We've tried disabling SSL. We've done multiple audits for leaked IDisposables. We've messed with ServicePointManager.DefaultConnectionLimit. The hangs still persist. Are others still experiencing this issue as well? |
|
Hey @tomlor, We just worked this issue by first: https://github.com/dotnet/corefx/issues/21796#issuecomment-381515493 And for downloading the blob data, we instead just generate a signed URL via the s3 client, and use HttpClient to download it instead of the s3 SDK. No issues now for 12 months. |
I have been fighting this issue for 6 months and still not good resolution. Help |
|
I think there is still an issue here. I'm strangely getting this issue in a ASP .NET Core 3.1 Web API project, but not getting it in a ASP .NET Core 3.1 MVC project. The code calling the GetObjectAsync task is exactly the same in both projects and using the same credentials and bucket. Been tearing my hair out all day, when this should be simple. |
|
@dyardyGIT @clevrdavid Depending on the stack trace you get when things get locked up, dotnet/runtime#22592 (comment) may fix this for you like it did for us. As for too many tasks, perhaps wrap your upload code path in a Polly Bulkhead code block which will help throttle the parallelism: https://github.com/App-vNext/Polly/wiki/Bulkhead |
|
I am shocked that there is not legitimate answer to this issue. I understand the 'randomness' makes it difficult but we have tried many things on both .net framework and .net core. This issue really was surfaced to a much greater extent when we made our move from Windows 2008 to Window 2012. On 2012 we have any to address http connection/socket issues. The comment about setting ServicePointManager.DefaultConnectionLimit=50 we have tried with no success. |
|
@dyardyGIT Disposing has certainly eliminated this problem for many. #152 (comment) |
Can you share how you generate an url using the sdk? thanks! |
|
Yes, it looks like this now, and still having the issue. |
|
Looks like our problem was mostly upload related, where it would hang, so we made our upload code just get the signed URL to upload to and PUT the data via regular HTTP calls: As for downloading, we do less of that, but you can imagine the flow is similar (client.GetPreSignedUrl), then just do a HTTP GET on that Url. |
I see you are disposing of the Stream and the GetObjectResponse, but not the AmazonS3Client. |
|
I'm a bit late to the party but I'd like to give my two cents on this issue. We're running .NET 4.7 on windows server 2016. We have multiple worker instances (some of which are running Quartz) and multiple instances running ASP.NET Web API. API instances never displayed any problems regarding S3 requests hanging, however, worker instances did. At first, I thought that we might have some kind of deadlock because logging in place wasn't verbose enough to pinpoint the exact location where the application stopped. It was clear that these hangs were happening only when the system was under a considerable load. Dumping the stack trace pointed me in the wrong direction. It showed that my service blocked on the following snippet executing I've tried everything I could find related to this issue including updating to .NET 4.8, using single instance S3 client, using different S3 clients for every unit of work, changing the existing implementation to use different SDK methods, etc. Tuning the This information steered me away from the problem source and into the esoteric search for the bug in the framework (hence the update to a newer version of the framework). While there might be cases where the framework bug caused your implementation to misbehave, that wasn't my case. Enter the following comment:
I inspected the facade implementation that we used to wrap S3 related actions and discovered that one of the Disposing of the object properly meant that no connections were left hanging and that the service could handle bigger loads without stopping. I still don't understand why the SDK lets this happen in the first place. I would've been happier with an exception telling me that the number of connections has been exceeded and that new connections cannot be established. |
|
I think I'm running into this issue as well. I'm trying to upload a 30kB text file from a .NET 4.7.2 desktop app with AWSSDK.S3 3.3.111.33. I run as a plugin in another app, which might be single-threaded. This line from the official docs never uploads the file and hangs until I force close the application. This happens every single time. I also see lingering I can only upload the file and close the request when I wrap the PutObjectAsync in a using statement, even though the only disposable object is the Task that it creates. But this is no longer async since there is no await in the PutObjectAsync method. So the UI becomes unresponsive. Any ideas on how to work around this? A second, related issue is that this behavior gets even worse when there is no internet connection. My working sample above times out at about 100 seconds. That's a long time to wait. I have found no setting that had any effect on this. |
|
@brinkdinges my workaround I'd suggest is to bypass the SDK for uploading, see my comment here #152 (comment) The workaround is to have the SDK generate the HTTPS Upload URL (that includes the signing key), then use HttpClient to PUT the data directly (pretty much what the SDK does, anyway). |
|
@brinkdinges your using statement is also wrong, you should await the task to ensure you are async: |
|
@Plasma Thank you. I had already tried to use your implementation, but I couldn't find what the SharedClient and Policy where. I found the policies in the SDK, but none that used generics. Could you clarify those two? PutObjectAsync indeed doesn't implement IDisposable. But without the using, the request never ends or gives a result. That is the reason I'm trying all these workarounds. I just tried GetObjectAsync and the same happens, it also never returns a result. Right now I have wrapped the whole method that creates credentials, the client and the request in a single task and I await that. This works for now. |
|
Ah, SharedClient is just an instance of HttpClient, and Policy is Polly.Net Retry Policy -- this part is kinda optional and you can skip it. But, your original question above, you had pasted code with a known anti-pattern of using async, where a deadlock can definitely occur, because you are not await'ing a task, but instead trying to access the public static async Task PutObjectAsync(AmazonS3Client client, PutObjectRequest putRequest)
{
var result = await client.PutObjectAsync(putRequest);
var success = result.HttpStatusCode == HttpStatusCode.OK;
if (!success) throw new CannotUploadFile();
} |
|
I am very sure that's what I started with. It didn't work then, it does now 😄 Thanks for pushing me to do it the correct way. |
and others
Randomly my application will hang infinity waiting for a response in the Amazon library for AmazonS3Client.GetOject see the stacktrace below, the object is not being reused.
here is a sample of the code run in that thread:
The text was updated successfully, but these errors were encountered: