This repository has been archived by the owner on Mar 3, 2023. It is now read-only.
Introduce stateful restorer in stmgr. #2091
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
all checkpoint save/restore as well as exactly once messages
nlu90
reviewed
Jul 21, 2017
| CHECK(stateful_restorer_); | ||
|
|
||
| // Start the restore process | ||
| stateful_restorer_->StartRestore(_checkpoint_id, _restore_txid, pplan_); | ||
| } | ||
|
|
||
| // Called by TmasterClient when it receives directive from tmaster | ||
| // to restore the topology to _checkpoint_id checkpoint |
There was a problem hiding this comment.
this comment here is duplicated with the one in the above method. Could you update it to correctly describe the actual behavior of this method?
There was a problem hiding this comment.
Fixed the comment
| @@ -587,6 +610,11 @@ const proto::system::PhysicalPlan* StMgr::GetPhysicalPlan() const { return pplan | |||
|
|
|||
| void StMgr::HandleStreamManagerData(const sp_string&, | |||
| proto::stmgr::TupleStreamMessage2* _message) { | |||
| if (stateful_restorer_ && stateful_restorer_->InProgress()) { | |||
There was a problem hiding this comment.
should we have some metrics for the amount of data being dropped for both stmgr data and instance data for future tracking and investigation?
| // in case we are not in 2pc | ||
| if (stateful_restorer_) { | ||
| if (!stateful_restorer_->InProgress() && tmaster_client_) { | ||
| LOG(INFO) << "We lost connection withi stmgr " << _stmgr_id |
| if (stateful_restorer_->InProgress()) { | ||
| // We are in the middle of a restore | ||
| stateful_restorer_->HandleAllInstancesConnected(); | ||
| } else if (tmaster_client_ && tmaster_client_->IsConnected()) { |
There was a problem hiding this comment.
could you elaborate why in this case the ResetTopologyState message needs to be sent?
There was a problem hiding this comment.
Changed the description
nlu90
approved these changes
Jul 24, 2017
nicknezis
pushed a commit
that referenced
this pull request
Sep 14, 2020
* Introduce stateful restorer in stmgr. This allows stmgr to handle all checkpoint save/restore as well as exactly once messages * Added metrics to keep track of bytes/tuples discarded during restore
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows stmgr to handle all checkpoint save/restore as well as exactly once messages.