Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[planner] support the distrubuted plan fragment #6

Closed
BohuTANG opened this issue Dec 20, 2020 · 0 comments
Closed

[planner] support the distrubuted plan fragment #6

BohuTANG opened this issue Dec 20, 2020 · 0 comments
Assignees
Labels
C-feature Category: feature prio: high High priority

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Dec 20, 2020

[summary]

For now, all the pipeline build based on the single fragment, if we have a query:

SELECT SUM(number) from system.numbers_mt

Thie pipelines building like:

  └─ AggregateFinalTransform × 1 processor
    └─ Merge (AggregatePartialTransform × 8 processors) to (MergeProcessor × 1)
      └─ AggregatePartialTransform × 8 processors
        └─ SourceTransform × 8 processors  

All 8-way parallel is executed on the local, so we need the plan can split to many fragments and some of them can be executed on the remote site:

  └─ AggregateFinalTransform × 1 processor
    └─ Merge (AggregatePartialTransform × 8 processors) to (MergeProcessor × 1)
      └─ AggregatePartialTransform@remote1 × 2 processors
        └─ SourceTransform@remote1 × 2 processors  
      └─ AggregatePartialTransform@remote2 × 2 processors
        └─ SourceTransform@remote2 × 2 processors  
      └─ AggregatePartialTransform@local × 4 processors
        └─ SourceTransform@local × 4 processors  

Local and remote stream is connected by the ExchangeStream port.

@BohuTANG BohuTANG self-assigned this Dec 20, 2020
@BohuTANG BohuTANG added the C-feature Category: feature label Dec 20, 2020
@BohuTANG BohuTANG added this to the alpha-0.1 milestone Dec 27, 2020
@BohuTANG BohuTANG added the prio: high High priority label Dec 27, 2020
@Xuanwo Xuanwo mentioned this issue Feb 14, 2022
6 tasks
BohuTANG pushed a commit that referenced this issue Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature Category: feature prio: high High priority
Projects
None yet
Development

No branches or pull requests

1 participant