[FLINK-8643] [flip6] Use JobManagerOptions#SLOT_REQUEST_TIMEOUT in ExecutionGraph #5475
+3,381
−1,400
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tillrohrmann
force-pushed
the
slotAllocationTimeout
branch
2 times, most recently
from
faa787c
to
ac83405
Compare
tillrohrmann
force-pushed
the
slotAllocationTimeout
branch
from
ac83405
to
b1cd8e3
Compare
tillrohrmann
force-pushed
the
slotAllocationTimeout
branch
2 times, most recently
from
b9873c0
to
19780c9
Compare
|
Merging this PR. |
tillrohrmann
force-pushed
the
slotAllocationTimeout
branch
from
19780c9
to
c360b20
Compare
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Feb 14, 2018
…ecutionGraph This commit changes the initialization of the ExecutionGraph to use the JobManagerOptions#SLOT_REQUEST_TIMEOUT for the slot allocation. Furthermore, it changes the behaviour of the SlotPool#ProviderAndOwner implementation such that the timeout is given to it via the SlotProvider#allocateSlot call. This closes apache#5475.
Let all Yarn entry points use the YarnConfigOptions.APPLICATION_MASTER_PORT option to specify the valid port range for the common RpcService. This closes apache#5388.
…n result retrieval Split RestClusterClient#submitJob into submitJob and requestJobResult which can be called individually. This closes apache#5428.
Register the JobTerminationHandler at the WebMonitorEndpoint to make it accessible to all REST endpoints. This closes apache#5429.
In order to support the job cancellation from the web UI, including when using Yarn, we have to register the JobTerminationHandler under /jobs/:jobid/yarn-cancel and /jobs/:jobid/yarn-stop. This is just a temporary fix until we can send arbitrary REST verbs through the Yarn proxy. This closes apache#5430.
The MiniDispatcher is responsible for submitting the single job with which a job mode cluster is started. Once the job has completed and if the cluster has been started in detached mode, the MiniDispatcher will terminate. In order to reduce code duplication, the MiniDispatcher is a sub class of the Dispatcher which is started with a single job submitted job graph store. This closes apache#5431.
This commit allows to deploy detached job mode clusters via the CliFrontend. In order to do that, it first extracts the JobGraph from the PackagedProgram and then uses the ClusterDescriptor to deploy the job mode cluster. This closes apache#5432.
Upon notification of newly allocated containers, the YarnResourceManager will only accept as many containers as there are pending container requests. All excess containers will be returned. This closes apache#5436.
…Services This closes apache#5458.
…luster This commit makes the RestClusterClient aware whether the user wishes to submit a job in detached or non-detached mode. If it is detached, then the RestClusterClient won't poll for the execution result. This closes apache#5466.
Shutting AkkaRpcActors down with Kill can result in uncompleted futures, if the mailbox contains requests which will be dropped. Therefore, it is better to stop AkkaRpcActors with a PoisonPill which assures that all messages before the pill will be processed. This closes apache#5476.
…ecutionGraph This commit changes the initialization of the ExecutionGraph to use the JobManagerOptions#SLOT_REQUEST_TIMEOUT for the slot allocation. Furthermore, it changes the behaviour of the SlotPool#ProviderAndOwner implementation such that the timeout is given to it via the SlotProvider#allocateSlot call. This closes apache#5475.
tillrohrmann
force-pushed
the
slotAllocationTimeout
branch
from
c360b20
to
341467d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This commit changes the initialization of the ExecutionGraph to use the
JobManagerOptions#SLOT_REQUEST_TIMEOUT for the slot allocation. Furthermore,
it changes the behaviour of the SlotPool#ProviderAndOwner implementation such
that the timeout is given to it via the SlotProvider#allocateSlot call.
Brief change log
SlotPool#ProviderAndOwnerwith slot allocation timeout. Instead require it as a parameter ofSlotProvider#allocateSlotJobManagerOptions#SLOT_REQUEST_TIMEOUTtoExecutionGraphas allocation timeoutVerifying this change
This change is already covered by existing tests, such as
SlotPoolTest.Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation