-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why does it take the ThreadPool 10+ seconds to shutdown when there is nothing running. #2450
Comments
By Contrast, the following will exit immediately when I manage Threads myself.
|
I have noticed something similar on windows (Foundation tests hangs for a while before completion) since we moved the develop branch (future 2.0 release) back-end thread implementation to c++11 std threading,events etc. Never had time to investigate, the only place I can see where ThreadPool would be doing it is PooledThread::release(), but I can't see why. If you find out the reason, let us know (or, even better, send a pull). |
The start of the issues is happening here via ~ThreadPool
This is one of those things that we need architectural feedback from the authors as to why before we can assert that something needs to be fixed. I can live with an answer like "This ThreadPool is not for you". |
What we have here is a reproducible hang on one platform (I did not notice it anywhere else except on windows). So, it is obvious it needs to be fixed. If you do not need ThreadPool, that's fine. If you do need it and want to fix it, then find why it hangs and let us know. Otherwise, it will be fixed when someone has enough time to spend on it. FWIW, I suspect it has to do with new event implementation, which uses std::condition_variable. I did some work on improving new events, but it did not fix the problem and I did not have time to look deeper into it; the contributors who ported threading and events to std have been silent since, so that's where we currently are. |
@aleks-f I am zeroing in on where the issue is. The root of the problem is that PoolThread, enters into a wait as shown here;
This happens after my initial runables are gone and that _pTarget is NULL. During shutdown, the following gets called for each PooledThread
Where _targetReady.set(); is called, and what is expected is that the void PooledThread::run() then wakes and goes away. This isn't happening, whereas the thinking it seems is that the release is signalling for a graceful shutdown to happen.
doesn't relinquish control. I need to dust off my mutex knowledge as to why this is not getting signaled. Jump in anytime ;) |
Probably nothing to do with mutexes. I'd suspect that perhaps the thread is not running when release() is called, I'm just guessing, though - best to walk in debugger and see what's really happening. |
Sorry, dust off my event knowledge. H |
More Clues.
Now it works as expected. More Experimentation... Lets try a std::unique_ptr
This doesn't work, even though the unique_ptr's destructor fired and deleted the contained ThreadPoolSingletonHolder pointer. There is something about tear-down order happening here. More Experimentation. Scope & Stack.
This works and the shutdown happens right away. This is where my knowledge of Windows x64 "what the hell is going on during the app being torn down" is hitting a wall. |
@aleks-f
I would expose the the objects so that they can be disposed of prior to app tear down. I don't know how far this goes, as POCO uses ThreadPool for other stuff. |
As is, anything that uses the default threadpool will have a shutdown problem on the Windows x64 builds I am testing.
|
So the theory that Events don't get signaled at tear-down, because they don't seem to wake up the PoolThread when the ThreadPool telling each PoolThread to release is false. Proof Project The reason there is a 10 second hang is that the PoolThread is waiting on an event, which gets set but the PoolThread doesn't wake. Why it works in my little POC, but not in the ThreadPool context. I don't believe this has to do with the condition work you mentioned. Here is the Poco.ThreadPool.Hang project that shows the hang. I will happily take any suggestions to hunt this down. |
This issue is stale because it has been open for 365 days with no activity. |
This issue is stale because it has been open for 365 days with no activity. |
Is this solved with #4311, perhaps? |
@matejk possible, but no time to check now, I'll mark this for 1.13.1 |
Reported problem was fixed with one of the changes for release 1.13. Added test case for this situation. |
* doc(ReleaseNotes): fix formatting, add PR links * Incorporated Debian patches (#4380) * Debian: Use null as device file as console might not be there * Debian: Add GNU Hurd support * Debian: Includes not available on Hurd * Debian: Disable SHA2 test on platforms where it's broken * Debian: Set POCO_NO_FPENVIRONMENT for armel --------- Co-authored-by: Jochen Sprickerhof <git@jochen.sprickerhof.de> * fix(UUID): UUID parser silently ignores too long strings #4375 (#4384) * fix(Crypto): EVP_CIPHER_CTX_init is incorrectly defined in Envelope.cpp if it is not defined already by OpenSSL. Fixed to properly use EVP_CIPHER_CTX_reset. * enh(ci): Add macos sanitizers job (#4313) * enh(ci): macOS thread sanitizer * enh(ci): macOS sanitize jobs for undefined and address. * fix(test): lock std:cerr to prevent data race in TCP server tests (reported by clang thread sanitizer) #4313 * fix(test): Use 96-bit IV with aes-256-gcm to fix (#4347): I/O error: error:1C800066:Provider routines::cipher operation failed * mingw compile and link improvements (#4019) (#4391) * fix(platform): MinGW Compile and link errors: undefined reference to `WinMain' * fix(platform): MinGW compile UUID tests (conflicting UUID defined as GUID in rpcdce.h via windows.h) * enh(DateTimeParser): option to cleanup input string before parsing (#569). * fix(CppUnit): do not install #4398 * fix(DataTest): do not install #4398 * chore(SingleSocketPoller): spelling * fix(MailMessage): Compare lowercase content disposition headers when reading parts (#3650). * chore(cmake): CppUnit Foundation dependency documentation; fix indentation * fix(SocketReactorTest): deadlock test intermittently hangs #4400 * gcc/clang (-fvisibility=hidden): corrections to compile and work properly (#4394) * fix(ActiveRecord): missing ActiveRecordLib_API definitions for clang/gcc. * fix(FPEnvironment): export FPEnvironmentImpl classes (#4393, #3331) * fix(Crypto): export *Impl classes used from inlines (#4393, #3331) * fix(Dynamic): explicitly instantiate and export Dynamic::Struct for string and int (-fvisibility=hidden) (#4393, #3331) * fix(JSON): explicitly instantiate and export SharedPtr for JSON::Array and JSON::Object (-fvisibility=hidden) (#4393, #3331) * enh(CMake): Set symbol visibility to hidden (#4393, #3331) * enh(configure): user c++17 standard for iphone, Darwin and ARM-Linux. * fix(UTF): explicitly instantiate and export 16 and 32-bit strings (-fvisibility=hidden) (#4393, #3331) * fix(RecordSet): make Extraction.h internal and instantiate RecordsSet::column template functions only for supported types. (-fvisibility=hidden) (#4393, #3331) * fix(UTF): fix explicitly instantiation on Windows (-fvisibility=hidden) (#4393, #3331) * enh(CMake): Add github jobs for macOS with visibility set to hidden (#4393, #3331) * fix(CppParser): Add missing declarations for CppParser_API (#4393, #3331) * enh(CMake): Enable more options in github jobs for macOS with visibility set to hidden (#4393, #3331) * fix(MongoDB): Add missing MongoDB_API (#4393, #3331) * Implemented automated network library initialization for Windows MinGW targets (#4402) * Implemented automated network library initialization for Windows MinGW/GCC targets * Using POCO_COMPILER_MINGW instead of __GNUC__ --------- Co-authored-by: Jesse Hoogervorst <jesse@deltaxlab.com> * fix(Thread_POSIX): qnx build error: 'prctl' was not declared in this scope #4404 * fix: NULL pointer strategy when setting rotation never #4411 Regression from 66e93f9. * fix(progen): add LanguageStandard (stdcpp17, stdc11); regenerate vs170 projects * Implement GetAdaptersAddresses API (#4419) * Upgrade from GetAdaptersInfo to GetAdaptersAddresses API. The code has been swapped back to a buffer of bytes because the data structure built by GetAdaptersAddresses is a linked list and the returned size is not a multiple of the IP_ADAPTERS_ADDRESSES struct. * Adding back Poco/UnWindows.h * Undoing indents. * test(ThreadPool): unit test for thread pool shutdown when no worker is running. (#2450) * enh: #4216: use std::string literals * enh: #3890: Get rid of SingletonHolder * enh(File): Linux, macOS: microsecond precision for file times (create and modification time). * enh(tests): Ability to enable/disable testing of deprecated functionality. (#4425) * fix(SSLManager): Fixed regression introduced in PR #4103, fixes #4421 * fix(LogFile): Unify flushing behaviour of WIN32 and STD implementation (#2443) * chore(buildwin): remove old vs versions from build and progen scripts; update documentation * chore(buildwin): remove leftover closing curly * enh(SQLite): SQLite FTS5 #4367 * Release 1.13.1: Update release notes, changelog, contributors, version files. (#4440) * Update CONTRIBUTORS * fix(CppParser): Documentation generation (some minor fixes, WiP) #4441 * feat(CppParser): C++11 attributes support * feat(PocoDoc): C++11 attributes support * chore(doc): Changelog and release notes formatting * fix(CppParser): parsing of function template parameters and namespace imports * fix: make headers parseable by CppParser/PocoDoc * fix(PocoDoc): add -DPOCO_DOC * fix(PocoDoc): postgres headers not found * fix(PocoDoc): libpq include path * fix(XML): #4443: Upgrade libexpat to 2.6.0 * doc: updated changelog --------- Co-authored-by: Günter Obiltschnig <guenter.obiltschnig@appinf.com> Co-authored-by: Matej Kenda <matejken@gmail.com> Co-authored-by: Jochen Sprickerhof <git@jochen.sprickerhof.de> Co-authored-by: Jesse Hoogervorst <hoogervorstjesse@gmail.com> Co-authored-by: Jesse Hoogervorst <jesse@deltaxlab.com> Co-authored-by: Aron Budea <aron.budea@collabora.com> Co-authored-by: Andrew Auclair <andrewauclair@users.noreply.github.com>
Expected behavior
ThreadPool's destructor should exist right away when there are no runnables anymore
Actual behavior
ThreadPool is in a self imposed 10-second holding pattern waiting for stuff that is already gone
The joinAll is immediatly satisfied, but the ThreadPool destructor doesn't let go until 10 seconds later.
Steps to reproduce the problem
POCO version
Latest
Compiler and version
Microsoft Visual Studio Enterprise 2017
Version 15.8.2
VisualStudio.15.Release/15.8.2+28010.2016
Microsoft .NET Framework
Version 4.7.03056
Installed Version: Enterprise
Visual C++ 2017 00369-90250-05476-AA291
Microsoft Visual C++ 2017
ADL Tools Service Provider 1.0
This package contains services used by Data Lake tools
Application Insights Tools for Visual Studio Package 8.13.10627.1
Application Insights Tools for Visual Studio
ASP.NET and Web Tools 2017 15.8.05077.0
ASP.NET and Web Tools 2017
ASP.NET Core Razor Language Services 15.8.31590
Provides languages services for ASP.NET Core Razor.
ASP.NET Web Frameworks and Tools 2017 5.2.60618.0
For additional information, visit https://www.asp.net/
Azure App Service Tools v3.0.0 15.8.05023.0
Azure App Service Tools v3.0.0
Azure Data Lake Node 1.0
This package contains the Data Lake integration nodes for Server Explorer.
Azure Data Lake Tools for Visual Studio 2.3.4000.4
Microsoft Azure Data Lake Tools for Visual Studio
Azure Functions and Web Jobs Tools 15.8.05023.0
Azure Functions and Web Jobs Tools
Azure Stream Analytics Tools for Visual Studio 2.3.4000.4
Microsoft Azure Stream Analytics Tools for Visual Studio
C# Tools 2.9.0-beta8-63208-01
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Common Azure Tools 1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.
Cookiecutter 15.8.18201.1
Provides tools for finding, instantiating and customizing templates in cookiecutter format.
Fabric.DiagnosticEvents 1.0
Fabric Diagnostic Events
GitHub.VisualStudio 2.5.2.2500
A Visual Studio Extension that brings the GitHub Flow into Visual Studio.
JavaScript Language Service 2.0
JavaScript Language Service
JavaScript Project System 2.0
JavaScript Project System
JetBrains ReSharper Ultimate 2018.1 Build 112.0.20180414.70444
JetBrains ReSharper Ultimate package for Microsoft Visual Studio. For more information about ReSharper Ultimate, visit http://www.jetbrains.com/resharper. Copyright © 2018 JetBrains, Inc.
Microsoft Azure HDInsight Azure Node 2.3.4000.4
HDInsight Node under Azure Node
Microsoft Azure Hive Query Language Service 2.3.4000.4
Language service for Hive query
Microsoft Azure Service Fabric Tools for Visual Studio 2.3
Microsoft Azure Service Fabric Tools for Visual Studio
Microsoft Azure Stream Analytics Language Service 2.3.4000.4
Language service for Azure Stream Analytics
Microsoft Azure Stream Analytics Node 1.0
Azure Stream Analytics Node under Azure Node
Microsoft Azure Tools 2.9
Microsoft Azure Tools for Microsoft Visual Studio 2017 - v2.9.10730.2
Microsoft Continuous Delivery Tools for Visual Studio 0.4
Simplifying the configuration of continuous build integration and continuous build delivery from within the Visual Studio IDE.
Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines
Microsoft Library Manager 1.0
Install client-side libraries easily to any web project
Microsoft MI-Based Debugger 1.0
Provides support for connecting Visual Studio to MI compatible debuggers
Microsoft Visual C++ Wizards 1.0
Microsoft Visual C++ Wizards
Microsoft Visual Studio Tools for Containers 1.1
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.
Microsoft Visual Studio VC Package 1.0
Microsoft Visual Studio VC Package
MLGen Package Extension 1.0
MLGen Package Visual Studio Extension Detailed Info
NuGet Package Manager 4.6.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.
ProjectServicesPackage Extension 1.0
ProjectServicesPackage Visual Studio Extension Detailed Info
Python 15.8.18201.1
Provides IntelliSense, projects, templates, debugging, interactive windows, and other support for Python developers.
Python - Django support 15.8.18201.1
Provides templates and integration for the Django web framework.
Python - IronPython support 15.8.18201.1
Provides templates and integration for IronPython-based projects.
Python - Profiling support 15.8.18201.1
Profiling support for Python projects.
R Tools for Visual Studio 1.3.40517.1016
Provides project system, R Interactive window, plotting, and more for the R programming language.
ResourcePackage Extension 1.0
ResourcePackage Visual Studio Extension Detailed Info
ResourcePackage Extension 1.0
ResourcePackage Visual Studio Extension Detailed Info
Snapshot Debugging Extension 1.0
Snapshot Debugging Visual Studio Extension Detailed Info
SQL Server Data Tools 15.1.61808.07020
Microsoft SQL Server Data Tools
Syntax Visualizer 1.0
An extension for visualizing Roslyn SyntaxTrees.
Test Adapter for Boost.Test 1.0
Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.
Test Adapter for Google Test 1.0
Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.
ToolWindowHostedEditor 1.0
Hosting json editor into a tool window
TypeScript Tools 15.8.20801.2001
TypeScript Tools for Microsoft Visual Studio
Visual Basic Tools 2.9.0-beta8-63208-01
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Visual C++ for Linux Development 1.0.9.27924
Visual C++ for Linux Development
Visual F# Tools 10.2 for F# 4.5 15.8.0.0. Commit Hash: c55dd2c3d618eb93a8d16e503947342b1fa93556.
Microsoft Visual F# Tools 10.2 for F# 4.5
Visual Studio Code Debug Adapter Host Package 1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio
Visual Studio Tools for CMake 1.0
Visual Studio Tools for CMake
Visual Studio Tools for Containers 1.0
Visual Studio Tools for Containers
Visual Studio Tools for Universal Windows Apps 15.0.28010.00
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.
Operating system and version
Windows 10
Other relevant information
x64
The text was updated successfully, but these errors were encountered: