Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: using barrier to initialize the source actor #3067

Merged
merged 8 commits into from
Jun 8, 2022

Conversation

shanicky
Copy link
Contributor

@shanicky shanicky commented Jun 8, 2022

What's changed and what's your intention?

This PR tries to use Barrier to initialize the Source Actor, replacing the original Protobuf field

When the create source statement is executed, the source_manager spawns a loop coroutine to poll the source with the corresponding SplitEnumerator and store the result in memory.

At the end of create materialized view, stream_manager executes run_command(Command::CreateMaterializedView) to generate an AddMutation Barrier to initialize the entire DAG, and this PR tries to put the split initialization of the source actor into this Barrier.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

@codecov
Copy link

codecov bot commented Jun 8, 2022

Codecov Report

Merging #3067 (0deec99) into main (35216eb) will decrease coverage by 0.01%.
The diff coverage is 28.96%.

@@            Coverage Diff             @@
##             main    #3067      +/-   ##
==========================================
- Coverage   73.51%   73.50%   -0.02%     
==========================================
  Files         732      732              
  Lines       99469    99526      +57     
==========================================
+ Hits        73122    73153      +31     
- Misses      26347    26373      +26     
Flag Coverage Δ
rust 73.50% <28.96%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../frontend/src/optimizer/plan_node/stream_source.rs 100.00% <ø> (ø)
src/meta/src/stream/test_fragmenter.rs 99.28% <ø> (-0.01%) ⬇️
src/stream/src/from_proto/source.rs 0.00% <ø> (ø)
src/stream/src/executor/mod.rs 48.63% <7.14%> (-3.96%) ⬇️
src/meta/src/stream/source_manager.rs 19.58% <14.65%> (+5.30%) ⬆️
src/meta/src/barrier/command.rs 75.59% <60.86%> (-3.66%) ⬇️
src/meta/src/stream/stream_manager.rs 68.78% <61.90%> (+1.44%) ⬆️
src/stream/src/executor/chain.rs 90.81% <100.00%> (+0.29%) ⬆️
src/stream/src/executor/dispatch.rs 76.94% <100.00%> (+0.12%) ⬆️
src/stream/src/executor/source.rs 77.74% <100.00%> (-0.18%) ⬇️
... and 6 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@shanicky shanicky marked this pull request as ready for review June 8, 2022 09:41
Copy link
Contributor

@tabVersion tabVersion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LSTM!!!

Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
@shanicky shanicky enabled auto-merge (squash) June 8, 2022 10:27
@shanicky shanicky merged commit bcb9ffa into main Jun 8, 2022
1 check passed
@shanicky shanicky deleted the peng/source-barrier-init branch June 8, 2022 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants