[ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan

[Hisoka-X]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

SeaTunnel engine will receive the logical plan sent by the client, and the engine needs to convert it into a physical plan that can be directly executed. Therefore, it is necessary to process the logical execution plan and generate a physical plan through conversion. The specific process is as follows:

1. Logical Plan
   
   Received the logical plan, we need to remove redundant Actions, and verifying the Schema(Transform2 and Transform 5 should be same)
2. Execution Plan
   
   While converting to an execution plan:

• Transforms need to be merged, and the basis for merging is whether the data will be split after the Transform.
• Convert Shuffle Action to Queue
• Convert to multi pipeline

3. Physical Plan
   
   We will split the Pipeline into separate executable tasks according to the degree of parallelism, also need add SourceSplitEnumerator and SinkAggregatedCommitter task
   After this, can send task to task execution service. Then task can run normally.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[2013650523]

hi, Excuse me,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?

[Hisoka-X]

hi, Excuse me,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?

1. The Queue will create when use shuffle transform (at now call partition transform). Used for data shuffle and change parallelism
2. The cache feature you mentioned will be added later