Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(pipeline): enrich core pipelines processors. #10098

Merged
merged 9 commits into from
Feb 17, 2023
Merged

feat(pipeline): enrich core pipelines processors. #10098

merged 9 commits into from
Feb 17, 2023

Conversation

RinChanNOWWW
Copy link
Collaborator

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Add three more types pipeline processors.

  • DuplicateProcessor: duplicate the data from one input port to two output ports.
  • ShuffleProcessor: reorder the data flow from input ports to output ports.
  • Multiple producers single consumer sinker (MpscSinker, AsyncMpscSinker): sinker with multiple input ports.

Intuitive graphs:

Duplicate processor:

         ┌───────────┐
         │           ├─────► b
b ──────►│ Duplicate │
         │           ├─────► b.clone()
         └───────────┘


ShuffleProcessor:

       ┌─────────┐
b0────►│         ├───►b0
       │         │
b1────►│         ├───►b2
       │ Shuffle │
b2────►│[0,2,1,3]├───►b1
       │         │
b3────►│         ├───►b3
       └─────────┘

MpscSinker, AsyncMpscSink:

            ┌─────────────────┐
input0─────►│                 │
            │                 │
input1─────►│    MpscSinker   │
        .   │        /        │
        .   │ AsyncMpscSinker │
        .   │                 │
inputN─────►│                 │
            └─────────────────┘

@vercel
Copy link

vercel bot commented Feb 17, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated
databend ⬜️ Ignored (Inspect) Feb 17, 2023 at 0:16AM (UTC)

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Feb 17, 2023
@RinChanNOWWW RinChanNOWWW marked this pull request as draft February 17, 2023 06:50
@RinChanNOWWW RinChanNOWWW marked this pull request as ready for review February 17, 2023 08:25
@RinChanNOWWW RinChanNOWWW mentioned this pull request Feb 17, 2023
Merged
@zhang2014 zhang2014 merged commit 5877c28 into datafuselabs:main Feb 17, 2023
@RinChanNOWWW RinChanNOWWW deleted the enrich-pipeline branch February 17, 2023 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ariesdevil ariesdevil ariesdevil approved these changes

@zhang2014 zhang2014 zhang2014 approved these changes

@sundy-li sundy-li Awaiting requested review from sundy-li

@dantengsky dantengsky Awaiting requested review from dantengsky

Assignees
No one assigned
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@RinChanNOWWW @ariesdevil @zhang2014