-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(state-store): Different actors should use the same table id #2057
Conversation
711067f
to
77b37ae
Compare
Codecov Report
@@ Coverage Diff @@
## main #2057 +/- ##
==========================================
- Coverage 70.66% 70.64% -0.03%
==========================================
Files 636 636
Lines 80718 80705 -13
==========================================
- Hits 57037 57011 -26
- Misses 23681 23694 +13
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
b0913f6
to
2b2da03
Compare
2b2da03
to
a9517a0
Compare
10d069b
to
880dfc2
Compare
880dfc2
to
4a7502a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Better to have a table_id
also in lookup executor and arrange executor.
What's changed and what's your intention?
Should resolve #2028 and #2032 . After this PR, different actors of same fragment should fix back to use same table id. And it do not introduce new round of StreamNode traverse.
Add a HashMap
table_ids_map
in fragmenter. (operator_id -> local table ids). It records the local table ids of each stateful operator. For cases that generate new operatorWhen serialize StreamNode, get local table id from
table_ids_map
via operator id. Add thestart_table_id
and we get global table id.There will be a two-phase rewrite. First rewrite only write local table id into
StreamNode
(Build Fragment). Second rewrite will write local table id + offset (global table id) intoStreamNode
(Build actors).Note that I do not use a enum to wrap TableId (Like
LocalFragmentId
,LocalActorId
) cuz i think it's a bit over-design.Welcome for suggestions.
Checklist
Refer to a related PR or issue link (optional)