[SPARK-54528][CONNECT] Close URLClassLoader eagerly to avoid OOM #53233
+41
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloud-fan
changed the title
vicennial
approved these changes
Nov 26, 2025
Contributor
vicennial
left a comment
vicennial
left a comment
There was a problem hiding this comment.
LGTM, thanks for the improvement!
Comment on lines
223
to
236
| // Close classloaders to release resources. | ||
| try { | ||
| state.replClassLoader match { | ||
| case urlClassLoader: URLClassLoader => | ||
| urlClassLoader.close() | ||
| logInfo(log"Closed replClassLoader (URLClassLoader) for evicted session " + | ||
| log"${MDC(SESSION_ID, state.sessionUUID)}") | ||
| case _ => | ||
| } | ||
| } catch { | ||
| case NonFatal(e) => | ||
| logWarning(log"Failed to close replClassLoader for session " + | ||
| log"${MDC(SESSION_ID, state.sessionUUID)}", e) | ||
| } |
Contributor
There was a problem hiding this comment.
I believe this is redundant since, for isolated sessions, the replClassLoader is always set to ExecutorClassLoader here (and thus, never hits the match case) since the URI is always set
dongjoon-hyun
approved these changes
Nov 26, 2025
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @cloud-fan .
Merged to master/4.1 for Apache Spark 4.1.0.
dongjoon-hyun
pushed a commit
that referenced
this pull request
Nov 26, 2025
### What changes were proposed in this pull request? In Spark Connect, every client session has its own class loader in the server side (both driver and executors), for isolation purpose. However, when closing the session, we don't close the class loader. We rely on GC to close the class loader and release the resource, but it's unpredicatable and not efficient. When sessions are being created frequently, OOM may happen. This PR fixes the issue by closing URLClassLoader eagerly when closing a session. It also adds a config so that we can tune the session cache size in the executor side to save memory. With this patch, OOM is gone. ### Why are the changes needed? avoid OOM ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Our internal CI service has limited memory and `SparkConnectJdbcDataTypeSuite` keeps failing due to OOM. This test suite creates and closes sessions back to back frequently, and GC can't catch up to release the class loaders. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor 2.1.36 Closes #53233 from cloud-fan/mem. Lead-authored-by: Wenchen Fan <wenchen@databricks.com> Co-authored-by: Wenchen Fan <cloud0fan@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit e264965) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In Spark Connect, every client session has its own class loader in the server side (both driver and executors), for isolation purpose. However, when closing the session, we don't close the class loader. We rely on GC to close the class loader and release the resource, but it's unpredicatable and not efficient. When sessions are being created frequently, OOM may happen.
This PR fixes the issue by closing URLClassLoader eagerly when closing a session. It also adds a config so that we can tune the session cache size in the executor side to save memory. With this patch, OOM is gone.
Why are the changes needed?
avoid OOM
Does this PR introduce any user-facing change?
no
How was this patch tested?
Our internal CI service has limited memory and
SparkConnectJdbcDataTypeSuitekeeps failing due to OOM. This test suite creates and closes sessions back to back frequently, and GC can't catch up to release the class loaders.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor 2.1.36