-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: scheduling splits when creating a materialized view #1788
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1788 +/- ##
==========================================
- Coverage 71.33% 71.22% -0.12%
==========================================
Files 605 605
Lines 78729 78931 +202
==========================================
+ Hits 56163 56217 +54
- Misses 22566 22714 +148
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
Signed-off-by: Shanicky Chen <shanicky@MacBook-Pro-M1-Pro.local>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
Signed-off-by: Peng Chen <peng@singularity-data.com>
0056de7
to
242255b
Compare
Signed-off-by: Peng Chen <peng@singularity-data.com>
todo!() | ||
} | ||
} | ||
let bootstrap_servers = properties.get(KAFKA_CONFIG_BROKER_KEY).ok_or_else(|| { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
consider make it a func in common/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!
This PR initially provides the ability to
create materialized view from source
When creating the materialized view, the fragmenter generates the corresponding fragment by slicing the graph, and each fragement generates the corresponding actor group by setting the parallelism, during which the stream source actor is found by
SourceNode
matching and a one-time split scheduling is performed,Split
is the abstraction of the external partition in theSource,
and when the actor is created later, it will start the subscription and consume according to the split information.The subsequent design will asynchronize the split assign and push the split assignment/addition asynchronously through config change barrier after the overall startup, so as to obtain the ability to add dynamic splits and shorten the time to create mv
Checklist
Refer to a related PR or issue link (optional)