[SPARK-53934][SQL][FOLLOWUP] Close Spark connect client connection and mark as closed #53236
+9
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vinodkc
changed the title
Contributor
Author
|
@dongjoon-hyun , Please review |
| try { | ||
| conn.spark.interruptOperation(operationId) | ||
| } catch { | ||
| case _: Exception => |
Member
There was a problem hiding this comment.
It would be great if we can match the exceptions explicitly. For example,
java.net.ConnectException
Member
There was a problem hiding this comment.
Can we just use the utility method Utils.closeQuietly?
nvm, this is fine
Contributor
There was a problem hiding this comment.
Can we just use the utility method
Utils.closeQuietly?
+1
pan3793
reviewed
Nov 27, 2025
| resultSet = null | ||
| } | ||
| closed = false | ||
| closed = true |
Contributor
Does #53233 help here? |
Contributor
We should fully understand the issue. Do you know where we fail to cancel an operation when the operation is already done? What kind of exception we throw? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR fixes flaky test failures in SparkConnectJdbcDataTypeSuite by adding defensive exception handling in SparkConnectStatement.close(). It also fixes a bug where the statement was not properly marked as closed.
Why are the changes needed?
The test
get binary type(and potentially others) was failing ~27% of the time with:Root Cause: During statement cleanup, close() calls interruptOperation() which makes a gRPC network call.
This fails when:
Does this PR introduce any user-facing change?
No. This only affects internal test reliability.
How was this patch tested?
Verified the fix follows the established pattern in SparkSession.close() (lines 679-690)
The existing test suite validates correct behavior
The exception handling only affects cleanup, not core functionality
Was this patch authored or co-authored using generative AI tooling?
No