test(spark): Fix flaky TestSparkFilterHelper by giving it its own SparkSession - #19356
Merged
voonhous merged 1 commit intoJul 23, 2026
Merged
Conversation
…rkSession testConvertInExpression calls org.apache.spark.sql.functions.expr(...), which lazily builds a SessionState on the active SparkSession. TestSparkFilterHelper extended HoodieSparkClientTestHarness directly without wiring @beforeeach initSparkContexts(), so it relied on a leaked active session from a prior test in the same Surefire fork and intermittently failed with "LiveListenerBus is stopped" when that session's context was already stopped. Add @BeforeEach/@AfterEach that create and stop the session, matching HoodieClientTestBase and the sibling TestHoodieDataSourceHelper.
danny0405
approved these changes
Jul 23, 2026
Collaborator
3 tasks
voonhous
enabled auto-merge (squash)
July 23, 2026 07:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
org.apache.hudi.TestSparkFilterHelper.testConvertInExpressionis intermittently flaky, failing withIllegalStateException: LiveListenerBus is stopped(wrapped inError while instantiating 'SessionStateBuilder'). It last surfaced across the spark3.3/3.4/3.5test-spark-java-tests-part1jobs on an unrelated Flink-only PR (#19354, CI run https://github.com/apache/hudi/actions/runs/29977957778). The test callsfunctions.expr(...), which needs an activeSparkSession, butTestSparkFilterHelperextendsHoodieSparkClientTestHarnessdirectly and never wiresinitSparkContexts()(that wiring lives inHoodieClientTestBase), so it relied on a leaked active session from a prior test in the same Surefire fork and failed whenever that session's context was already stopped.Summary and Changelog
Add
@BeforeEach/@AfterEachthat call the inheritedinitSparkContexts()/cleanupSparkContexts()so the test owns a freshSparkSession, matchingHoodieClientTestBaseand the siblingTestHoodieDataSourceHelper. Test-only; no production code or assertions change. No code was copied.Impact
None. Test-only change confined to
TestSparkFilterHelper.Risk Level
none
Documentation Update
none
Contributor's checklist