test: fix SparkToColumnar plan-shape assertions on Spark 4#4032
Merged
andygrove merged 1 commit intoapache:mainfrom Apr 22, 2026
Merged
test: fix SparkToColumnar plan-shape assertions on Spark 4#4032andygrove merged 1 commit intoapache:mainfrom
andygrove merged 1 commit intoapache:mainfrom
Conversation
Closes apache#4031. Spark 4 wraps the final AQE plan in `ResultQueryStageExec`, a `LeafExecNode` that `SparkPlan.collect` treats as opaque. The two affected tests used `adaptivePlan.collect { case c: CometSparkToColumnarExec => c }`, which therefore found zero matches on Spark 4 even though the node was present in the materialized plan. Switch to the `collect` method from `AdaptiveSparkPlanHelper` (already mixed into `CometTestBase`), which descends through query stages. Drop the `assume(!isSpark40Plus)` guards that disabled the tests on Spark 4.
Member
Author
|
Merged. Thanks @parthchandra |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4031.
Rationale for this change
Two tests in
CometExecSuite(SparkToColumnar eliminate redundant in AQEandSparkToColumnar override node name for row input) were skipped on Spark 4 viaassume(!isSpark40Plus). When the guards are removed they both fail withList() had length 0 instead of expected length 1: the tests look for exactly oneCometSparkToColumnarExecin the final AQE plan and find zero.The
CometSparkToColumnarExecinsertion is actually working correctly. Spark 4 introducedResultQueryStageExec, which extendsLeafExecNodeand now wraps the final plan exposed byAdaptiveSparkPlanExec.executedPlan. The tests usedSparkPlan.collect, which treats aLeafExecNodeas opaque and does not descend intoQueryStageExec.plan, so the node was invisible to the assertion.What changes are included in this PR?
collectfromAdaptiveSparkPlanHelper(already mixed intoCometTestBase). The helper'scollectdescends throughAdaptiveSparkPlanExecandQueryStageExec.planviaallChildren, so it works identically on Spark 3.4 / 3.5 (where the top-level plan is not wrapped in aResultQueryStageExec) and Spark 4.assume(!isSpark40Plus)guards that were disabling the tests on Spark 4.How are these changes tested?
The two previously skipped tests now run and pass on Spark 4. Verified with:
All 9
SparkToColumnar*tests pass. Also verified on the default Spark 3.5 profile; all 9 still pass.