[FLINK-8611] [flip6] Add result future to JobManagerRunner #5434
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tillrohrmann
force-pushed
the
jobManagerRunnerFuture
branch
from
3c41468 to
ae5542f
Compare
Let all Yarn entry points use the YarnConfigOptions.APPLICATION_MASTER_PORT option to specify the valid port range for the common RpcService. This closes apache#5388.
…n result retrieval Split RestClusterClient#submitJob into submitJob and requestJobResult which can be called individually. This closes apache#5428.
Register the JobTerminationHandler at the WebMonitorEndpoint to make it accessible to all REST endpoints. This closes apache#5429.
In order to support the job cancellation from the web UI, including when using Yarn, we have to register the JobTerminationHandler under /jobs/:jobid/yarn-cancel and /jobs/:jobid/yarn-stop. This is just a temporary fix until we can send arbitrary REST verbs through the Yarn proxy. This closes apache#5430.
The MiniDispatcher is responsible for submitting the single job with which a job mode cluster is started. Once the job has completed and if the cluster has been started in detached mode, the MiniDispatcher will terminate. In order to reduce code duplication, the MiniDispatcher is a sub class of the Dispatcher which is started with a single job submitted job graph store. This closes apache#5431.
This commit allows to deploy detached job mode clusters via the CliFrontend. In order to do that, it first extracts the JobGraph from the PackagedProgram and then uses the ClusterDescriptor to deploy the job mode cluster. This closes apache#5432.
Upon notification of newly allocated containers, the YarnResourceManager will only accept as many containers as there are pending container requests. All excess containers will be returned. This closes apache#5436.
…Services This closes apache#5458.
…luster This commit makes the RestClusterClient aware whether the user wishes to submit a job in detached or non-detached mode. If it is detached, then the RestClusterClient won't poll for the execution result. This closes apache#5466.
Shutting AkkaRpcActors down with Kill can result in uncompleted futures, if the mailbox contains requests which will be dropped. Therefore, it is better to stop AkkaRpcActors with a PoisonPill which assures that all messages before the pill will be processed. This closes apache#5476.
…ecutionGraph This commit changes the initialization of the ExecutionGraph to use the JobManagerOptions#SLOT_REQUEST_TIMEOUT for the slot allocation. Furthermore, it changes the behaviour of the SlotPool#ProviderAndOwner implementation such that the timeout is given to it via the SlotProvider#allocateSlot call. This closes apache#5475.
This commit introduces a JobMasterConfiguration which contains JobMaster specific configuration settings. This closes apache#5478.
The JobMaster no longer needs to implement the RestfulGateway. Therefore, it is removed by this commit. This closes apache#5433.
This commit adds a CompletableFuture<ArchivedExecutionGraph> to the JobManagerRunner. This future will be completed once the job has reached a globally terminal state. This closes apache#5434.
tillrohrmann
force-pushed
the
jobManagerRunnerFuture
branch
from
ae5542f to
34d53ea
Compare
Contributor
Author
|
Rebased onto the soon to be master. |
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Feb 15, 2018
This commit adds a CompletableFuture<ArchivedExecutionGraph> to the JobManagerRunner. This future will be completed once the job has reached a globally terminal state. This closes apache#5434.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This commit adds a CompletableFuture to the
JobManagerRunner. This future will be completed once the job has
reached a globally terminal state.
This PR is based on #5431.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation