[scheduling] implement backtracking with intermediate result checking #595
Conversation
There was a problem hiding this comment.
we need the producing execution attempt id as well to uniquely identify a result partition
There was a problem hiding this comment.
The intend here is to lock any available ResulPartition for an IntermediateResultPartitionID, regardless of the ExecutionAttemptID.
|
@mxm could you add more information about this backtracking mode? Is this to support recovery of failed job? |
|
Yes @hsaputra. One of the goals is to support resume/recovery of a job from an intermediate result already produced. Another goal is to rework the deploy logic of the The changes in this pull request are a first step towards these goals. The |
|
Thanks @mxm, appreciate the additional information about the PR |
There was a problem hiding this comment.
Why do theses private interfaces (ScheduleActiona and PostBacktrackingHook) being added here?
There was a problem hiding this comment.
Absolutely right. I was going to change their location before merging. They will probably go into a new class dedicated for the backtracking alongside with the scheduleUsingBacktracking method. I had trouble finding the right spot and decided to leave them there for now.
|
@hsaputra You're welcome. Thanks for the additional feedback. |
mxm
force-pushed
the
backtracking-scheduling
branch
2 times, most recently
from
4c41ecb to
51edf02
Compare
- backtracks from the sinks of an ExecutionGraph - checks the availability of IntermediatePartitionResults - marks ExecutionVertex to be scheduled This first version of backtracking does not support resume/recovery from intermediate results yet. It lays the foundation for integrating the remaining changes.
mxm
force-pushed
the
backtracking-scheduling
branch
from
51edf02 to
027d4fd
Compare
|
I'm closing because I opened a new pull request #640. |
…ory (apache#595) This change removes cc-flink-managed-table module from flink repository. Also, remove its references in this repository. This module is being moved to kdb repository. Also, removed CompiledPlanJobGraphGeneratorV3ImplTest. It fails currently because we are removing cc-flink-managed-table module as part of this PR and this test depends on that module. The test is also present at cc-flink-docker/cc-flink-runtime-integration-impl module.
3 participants
This first version of backtracking does not support resume/recovery from
intermediate results yet. It lays the foundation for integrating the
remaining changes.