[SPARK-55317] Add SequentialUnion logical plan node and planning rule #54098
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add SequentialUnion logical plan node for sequential source processing. Unlike Union which processes all children concurrently, SequentialUnion processes each child source sequentially, enabling backfill-to-live streaming scenarios. Key features: - Extends UnionBase for schema compatibility - Supports flatten operation for chaining - Validates minimum 2 children - Compatible with byName and allowMissingCol parameters This is the first step in implementing Sequential Union support, providing the logical plan foundation for the execution engine.
ericm-db
force-pushed
the
sequential-union-logical-node
branch
2 times, most recently
from
0235299 to
de63f31
Compare
### What changes were proposed in this pull request? This PR introduces the SequentialUnion logical plan node and associated analysis rules for sequential processing of streaming sources. Changes: 1. Added SequentialUnion logical plan node extending UnionBase 2. Added companion object with apply() and flatten() methods 3. Implemented FlattenSequentialUnion analysis rule 4. Implemented ValidateSequentialUnion analysis rule 5. Added error classes for validation failures 6. Registered analysis rules in Analyzer 7. Added comprehensive test suite for analysis rules ### Why are the changes needed? This provides the foundational infrastructure for Sequential Union support, which enables backfill-to-live streaming scenarios. The analysis rules ensure proper plan structure (flattening nested unions) and validate constraints (all streaming sources, no nesting after flattening). ### Does this PR introduce any user-facing change? No. This PR only adds internal logical plan infrastructure and analysis rules. No user-facing APIs are introduced. ### How was this patch tested? Added SequentialUnionAnalysisSuite with tests for: - Flattening logic (single level, nested, deeply nested, multiple nested) - Validation of streaming requirements - Validation of nesting constraints - Error message verification using checkError framework ### Was this patch authored or co-authored using generative AI tooling? No.
ericm-db
force-pushed
the
sequential-union-logical-node
branch
from
de63f31 to
555fcdf
Compare
ericm-db
changed the title
JIRA Issue Information=== Task SPARK-55317 === This comment was automatically generated by GitHub Actions |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR introduces the SequentialUnion logical plan node, which enables sequential processing of multiple streaming sources. Unlike the existing Union operator that processes all children concurrently, SequentialUnion processes each child source to completion before moving to the next.
Key changes:
This is the foundational logical plan component. Execution support and user-facing APIs will be added in follow-up PRs.
Why are the changes needed?
Sequential Union enables backfill-to-live streaming scenarios where historical data must be processed completely before transitioning to live data, while preserving stateful operator state across the transition.
Example use case: Process historical Parquet data completely, then seamlessly switch to live Kafka streaming - all within a single query that maintains aggregations, watermarks, and deduplication state.
This pattern is not currently possible with the existing Union operator, which processes all sources concurrently.
Does this PR introduce any user-facing change?
No. This PR only adds the internal logical plan node. No user-facing APIs or behavior changes are introduced yet.
How was this patch tested?
Added new test suite SequentialUnionSuite with tests covering:
Was this patch authored or co-authored using generative AI tooling?
No