Skip to content

[GLUTEN-11920][CORE] Keep columnar children for dual-mode parents with row output#12004

Merged
zhztheplayer merged 2 commits into
apache:mainfrom
Yao-MR:bugfix_dual-mode-fix
May 8, 2026
Merged

[GLUTEN-11920][CORE] Keep columnar children for dual-mode parents with row output#12004
zhztheplayer merged 2 commits into
apache:mainfrom
Yao-MR:bugfix_dual-mode-fix

Conversation

@Yao-MR
Copy link
Copy Markdown
Contributor

@Yao-MR Yao-MR commented Apr 28, 2026

What changes are proposed in this pull request?

For dual-mode parents that support both columnar and row-based execution, Gluten previously let the child convention follow the parent's output convention by default. When the parent was required to produce rows, this could incorrectly push a row requirement down to its children even though the parent could still consume batch input and satisfy the row output requirement by itself.

This PR changes dual-mode parents to keep a batch child convention under a row output requirement, which aligns transition insertion with Spark behavior and avoids unnecessary row transitions.

The change is applied in both the plan-side child convention derivation and the transition convention derivation so the behavior stays consistent.

This PR also:

  • adds transition tests to verify that a dual-mode parent keeps a columnar child for row output;
  • adds coverage to verify that a row child is converted to batch when required by a dual-mode parent; and
  • re-enables GlutenColumnarRulesSuite on Spark 4.1.

How was this patch tested?

  • Added transition tests for dual-mode parent convention handling under row output requirements.
  • Re-enabled GlutenColumnarRulesSuite on Spark 4.1 to cover the fixed behavior.

issue: #11920

fix in spark: apache/spark#50239

Was this patch authored or co-authored using generative AI tooling?

@github-actions github-actions Bot added the CORE works for Gluten Core label Apr 28, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@Yao-MR Yao-MR changed the title 【Gluten-11920]: Keep columnar children for dual-mode parents with row… 【Gluten-11920]: Keep columnar children for dual-mode parents with row output Apr 28, 2026
@Yao-MR Yao-MR marked this pull request as ready for review April 28, 2026 12:15
@Yao-MR Yao-MR changed the title 【Gluten-11920]: Keep columnar children for dual-mode parents with row output [Gluten-11920]: Keep columnar children for dual-mode parents with row output Apr 28, 2026
@zhouyuan zhouyuan changed the title [Gluten-11920]: Keep columnar children for dual-mode parents with row output [GLUTEN-11920]: Keep columnar children for dual-mode parents with row output Apr 29, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@FelixYBW FelixYBW requested a review from zhztheplayer May 6, 2026 02:11
Copy link
Copy Markdown
Contributor

@zml1206 zml1206 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@zml1206 zml1206 changed the title [GLUTEN-11920]: Keep columnar children for dual-mode parents with row output [GLUTEN-11920][CORE] Keep columnar children for dual-mode parents with row output May 8, 2026
@zhztheplayer zhztheplayer merged commit 8d76cb8 into apache:main May 8, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants