Reorder teardown and add a post-exit hook registry #6213
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The order here was problematic already due to Timeout registering its shutdown hook alongside other `at_exit` hooks, but I realized while fixing jruby#6212 that there are other issues: * The ThreadService is shut down before user-defined finalizers have run. * The SIGEXIT tweaking happens before some user code has run. * Services that have internal shut down hooks (like Timeout) have no way to register a non-finalizer exit hook without potentially interfering with user-defined exit hooks. This commit reorders teardown logic to avoid interfering with user code and adds a mechanism to register "post exit" shutdown hooks.
This appears to interfere with other services shutting down, like Timeout's executor. Anything that might try to access JRuby thread state should happen before this.
headius
added a commit
that referenced
this pull request
May 12, 2020
There's not a lot of context on the commit from 2013 that added this, but I suspect it's not really correct behavior anymore. In the process of reordering teardown (#6213) this logic began to cause a spec hang for Kernel.exit! specs due to the teardown deadlocking with the main thread: * Main thread starts up and joins a thread that calls exit!. The java.lang.Thread.join call locks the thread object. * Thread calls exit!, which calls System.exit (as opposed to the non-bang exit, which terminates the main thread and lets it tear things down safely). * System exit calls this teardown hook, which attempts to shut terminate and join remaining threads. * Because the teardown is not called from the main thread, the system deadlocks trying to shut down the child thread, which is locked in the main thread doing join. The chain of deadlock is main => thread 1 join => system exit => thread 2 for teardown => try to kill and join thread 1. The ultimate deadlock is due to a circular sequence of calls that can't be interrupted (System.exit's wait on the teardown thread) combined with the teardown happening from a non-main thread. It's not clear why this started to hang after the reordering. Moving the ThreadService teardown back to its former relative position does not fix the hang, and logically I don't know why it would... there should still be a deadlock if main is joining a thread and the System exit hook tries to tear that thread down.
headius
added a commit
to headius/jruby
that referenced
this pull request
Sep 17, 2020
This is a cherry-pick of the crucial commit from jruby#6213 that should also fix jruby#6379, where users reported that at_exit and other "clean shutdown" hooks were improperly running during a "hard shutdown" using `exit!`. The original commit message for c941c41 follows: Remove JVM shutdown hook for teardown There's not a lot of context on the commit from 2013 that added this, but I suspect it's not really correct behavior anymore. In the process of reordering teardown (jruby#6213) this logic began to cause a spec hang for Kernel.exit! specs due to the teardown deadlocking with the main thread: * Main thread starts up and joins a thread that calls exit!. The java.lang.Thread.join call locks the thread object. * Thread calls exit!, which calls System.exit (as opposed to the non-bang exit, which terminates the main thread and lets it tear things down safely). * System exit calls this teardown hook, which attempts to shut terminate and join remaining threads. * Because the teardown is not called from the main thread, the system deadlocks trying to shut down the child thread, which is locked in the main thread doing join. The chain of deadlock is main => thread 1 join => system exit => thread 2 for teardown => try to kill and join thread 1. The ultimate deadlock is due to a circular sequence of calls that can't be interrupted (System.exit's wait on the teardown thread) combined with the teardown happening from a non-main thread. It's not clear why this started to hang after the reordering. Moving the ThreadService teardown back to its former relative position does not fix the hang, and logically I don't know why it would... there should still be a deadlock if main is joining a thread and the System exit hook tries to tear that thread down.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the following problems:
at_exithook, potentially breakingat_exithooks that were registered earlier. (Timeout service shuts down before at_exits have run #6212)