fix: attach tokio runtime threads to the JVM as daemon threads - #5748
Open
zhangfengcdt wants to merge 2 commits into
Open
fix: attach tokio runtime threads to the JVM as daemon threads#5748zhangfengcdt wants to merge 2 commits into
zhangfengcdt wants to merge 2 commits into
Conversation
jni-rs attaches runtime threads lazily as non-daemon JVM threads, and DestroyJavaVM waits for those before running the shutdown hook that would release the runtime. An application that returns from main without SparkContext.stop() therefore never exits. Attach each runtime thread with AttachCurrentThreadAsDaemon on start and detach it on stop, so the shutdown hook can run and release the runtime. Add a forked-JVM regression test and update the threading section of the contributor guide. Closes apache#5664
zhangfengcdt
force-pushed
the
fix/daemon-runtime-threads
branch
from
September 6, 2026 18:41
7c0e46d to
7f127af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5664.
Rationale for this change
#4734 releases the tokio runtime from the driver/executor plugin
shutdown(), which fixes theJVM exit hang whenever
SparkContext.stop()is called. It does not help when an applicationsimply returns from
main(or a PySpark driver exits) without callingspark.stop():acquireMemory, scans fed from JVMiterators, scalar subqueries, ...) are attached lazily by jni-rs with
AttachCurrentThread,which makes them non-daemon JVM threads.
DestroyJavaVMwaits until all non-daemon threads have exited before running shutdown hooks.SparkContext.stop()-> pluginshutdown()->release_runtime(), so the threads can only be released by a hook that can only run after thethreads have exited. The JVM hangs forever.
Spark's own thread pools are daemon threads for exactly this reason.
What changes are included in this PR?
build_runtimeregisters tokioon_thread_start/on_thread_stophooks that attach eachruntime thread with
AttachCurrentThreadAsDaemonand detach it before the thread exits. jni-rsreuses an existing attachment, so its lazy non-daemon attach is never triggered on these
threads. jni-rs intentionally offers no daemon attach API, hence the raw invoke-interface
calls. The hooks are no-ops when no JVM is registered (Rust unit tests and benchmarks).
SparkContext.stop()is called.How are these changes tested?
New
CometRuntimeShutdownSuiteforks a child JVM that runs a native Comet query with the unifiedmemory pool (which calls into the JVM from a tokio worker) and returns from
mainwithoutSparkContext.stop(). The parent asserts the child exits with status 0 within a deadline. Achild JVM is required because the failure mode is "the JVM cannot exit", which cannot be observed
from inside that JVM.
Without the fix the child prints its marker and never exits (killed at the deadline); with the
fix it exits in a few seconds. Also verified
CometPluginsSuite(theSparkContext.stop()path)and the native planner unit tests, which build the runtime without a JVM.