[WIP] Implement join #559
[WIP] Implement join #559
Conversation
| } | ||
|
|
||
| async fn execute(&self) -> Result<SendableDataBlockStream> { | ||
| let read_left_task = tokio::task::spawn(Self::read_from_sink(self.left.clone())); |
There was a problem hiding this comment.
tokio:task::spawn will push the task to the global tokio runtime.
In order to limit the resources of one session, we should use context::execute_task:
https://github.com/datafuselabs/datafuse/blob/master/fusequery/query/src/pipelines/processors/processor_merge.rs#L64
It will bind all the threads used by the processors to the context's runtime.
There was a problem hiding this comment.
Thanks for reminding me about this.
I was intending to implement a really simple version at first to help me understand the whole procedure, but so far I still haven't find a proper way to build join transform into Pipeline.
There was a problem hiding this comment.
planner -> transform:
https://github.com/datafuselabs/datafuse/blob/master/fusequery/query/src/pipelines/processors/pipeline_builder.rs
For a sql, steps to a pipeline:
- parser to planner
- optimizer
- planner -> transform by pipeline builder
- executor the pipeline
There was a problem hiding this comment.
@BohuTANG Yes, but it seems fusequery currently has strong consumption that a PlanNode only has one input, which doesn't fit join well.
I'm finding a way to handle this properly.
There was a problem hiding this comment.
We can extend it to many if need, processor now has many inputs.
leiysky
force-pushed
the
feature/support-parse-join
branch
from
946bd8b
to
78b5503
Compare
github-actions
bot
added
C-documentation
leiysky
force-pushed
the
feature/support-parse-join
branch
from
78b5503
to
4766d2d
Compare
leiysky
force-pushed
the
feature/support-parse-join
branch
from
4766d2d
to
b653b0b
Compare
databend-bot
added
the
pr-feature
|
Thanks for the contribution! Please review the labels and make any necessary changes. |
|
We have cartesian product now. |
|
I will finish the prequisite of Join in other PRs first, so this PR won't be ready for review within a predictable period of time. |
Summary
Initial tasks to implement join:
PipelineChangelog
Related Issues
Related to #319
Test Plan
Unit Tests
Stateless Tests