Deadlock on calling Aws::ShutdownAPI via s_registryMutex #3282
Comments
Markus87
added
bug
|
do you have reproduction code? how frequently do you see the issue? what version of the SDK are you using and how to you install it? |
sbiscigl
added
the
response-requested
|
Sadly I can't reproduce it (yet). |
github-actions
bot
removed
the
response-requested
|
So i think the issue can be summarized by the error log (that i assume you have in your logs) which essentially means yes, the sdk is still processing async requests while being shutdown. this "FATAL" log should be a an exception if we had exceptions enabled. the solution for this would be in your client code to call the ShutdownSdkClient method on the underlying s3 client with a non-negtive time out time, to wait for processing to be completed before destorying the client. |
sbiscigl
added
the
response-requested
|
Thanks for the feedback. So if I have understood you correctly, even if the S3Client does call Documentation says If there is anything I can check in the dump for you please just ask. |
github-actions
bot
removed
the
response-requested
|
Hi @Markus87 , Thank you a lot for reporting this issue! Could you please provide a high-level code of how you use TransferManager? Especially for the part of
My guess is that you Submit a DownloadFile task but don't wait for it's completion. And there is a design issue for the existing implementation of TransferManager: TransferTask self-sustains the TransferManager pointer on the worker thread, resulting in a cyclic reference of I'd suggest to also call Best regards, |
|
Hello @SergeyRyabinin and thanks for your response. The code in question looks like this: class AWS::S3
{
//...
std::shared_ptr< Aws::S3::S3Client > _;
void Download(...)
{
auto executor = Aws::MakeShared<Aws::Utils::Threading::DefaultExecutor>( ALLOCATION_TAG );
Aws::Transfer::TransferManagerConfiguration config( executor.get() );
config.s3Client = _;
auto transfer = Aws::Transfer::TransferManager::Create( config );
auto transferHandle = transfer->DownloadFile( ... );
transferHandle->WaitUntilFinished();
if( transferHandle->GetStatus() != Aws::Transfer::TransferStatus::COMPLETED )
Throw( "Aws::Transfer::DownloadFile", transferHandle->GetLastError() );
}
//...
};
class AWS::VersionChannel
{
//...
std::shared_ptr< AWS::S3 > myS3;
void Download(...)
{
myS3->Download(...);
}
//...
};
AWS::VersionChannel GetVersionChannel()
{
return AWS::VersionChannel(...);
}
void MayDeadlockAfterDownload()
{
//...
GetVersionChannel().Download(...);
//...
}To avoid confusion, the If I follow your suggestion then we need to replace |
SergeyRyabinin
added
the
pending-release
|
Hi @Markus87 , returns on successful file transfer, however, the transfer task may be still active and be finishing on a separate thread, then at the same time your Download function returns, destroying the Executor/TransferManager and rendering the async transfer task to be the only owner of itself.
Best regards, |
|
@SergeyRyabinin Thank you, we will try that. |
Describe the bug
It seems the issue does not always occur.
Aws::ShutdownAPI holding s_registryMutex and joining thread who needs same mutex to finish
Thread with ~AmazonWebServiceRequest() waiting for s_registryMutex
Regression Issue
Expected Behavior
Aws::ShutdownAPI should complete without deadlocking.
Current Behavior
Aws::ShutdownAPI never completes.
Reproduction Steps
No example to reproduce available.
Possible Solution
No response
Additional Information/Context
No response
AWS CPP SDK version used
1.11.428
Compiler and Version used
msvc 19.42.34435 x64
Operating System and version
Windows Server 2019 / 1809 / 17763.6532
The text was updated successfully, but these errors were encountered: