Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(meta): support DAG fragment graph #1789

Merged
merged 2 commits into from
Apr 13, 2022
Merged

Conversation

skyzh
Copy link
Contributor

@skyzh skyzh commented Apr 12, 2022

Signed-off-by: Alex Chi iskyzh@gmail.com

What's changed and what's your intention?

Previously, we only supports a tree-like fragment graph. Now we refactor it to support any DAG.

  • Use topology sort to construct actors from fragments.
  • Fully support multi-dispatcher per fragment / per actor.
  • Differentiate local actor id and global actor id, generate all ids in batch.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

close #1751

Signed-off-by: Alex Chi <iskyzh@gmail.com>
@skyzh skyzh force-pushed the skyzh/refactor-fragmenter branch from 30363b4 to 07e369a Compare April 13, 2022 05:48
@skyzh skyzh marked this pull request as ready for review April 13, 2022 05:48
Signed-off-by: Alex Chi <iskyzh@gmail.com>
Copy link
Contributor

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LSTM!

@codecov
Copy link

codecov bot commented Apr 13, 2022

Codecov Report

Merging #1789 (18b5bed) into main (6b68bf1) will decrease coverage by 0.00%.
The diff coverage is 92.35%.

@@            Coverage Diff             @@
##             main    #1789      +/-   ##
==========================================
- Coverage   71.33%   71.33%   -0.01%     
==========================================
  Files         605      605              
  Lines       78729    79168     +439     
==========================================
+ Hits        56163    56474     +311     
- Misses      22566    22694     +128     
Flag Coverage Δ
rust 71.33% <92.35%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/meta/src/rpc/service/ddl_service.rs 0.00% <0.00%> (ø)
src/meta/src/rpc/service/stream_service.rs 0.00% <0.00%> (ø)
src/meta/src/model/mod.rs 95.58% <76.08%> (-2.84%) ⬇️
src/meta/src/stream/graph/stream_graph.rs 70.31% <92.41%> (+12.04%) ⬆️
src/meta/src/stream/fragmenter.rs 96.20% <95.00%> (+3.16%) ⬆️
src/meta/src/stream/graph/fragment_graph.rs 96.94% <96.36%> (-0.32%) ⬇️
src/meta/src/stream/test_fragmenter.rs 99.60% <100.00%> (+<0.01%) ⬆️
src/sqlparser/src/ast/statement.rs 82.87% <0.00%> (-4.80%) ⬇️
src/meta/src/stream/stream_manager.rs 72.50% <0.00%> (-3.26%) ⬇️
src/frontend/src/handler/create_source.rs 97.45% <0.00%> (-1.87%) ⬇️
... and 25 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@skyzh skyzh enabled auto-merge (squash) April 13, 2022 06:00
@skyzh skyzh merged commit 5575b70 into main Apr 13, 2022
@skyzh skyzh deleted the skyzh/refactor-fragmenter branch April 13, 2022 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

meta: refactor fragmenter to have full support for DAG
2 participants