Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-8613] [flip6] [yarn] Return excess containers #5436

Closed

Conversation

tillrohrmann
Copy link
Contributor

What is the purpose of the change

Upon notification of newly allocated containers, the YarnResourceManager
will only accept as many containers as there are pending container requests.
All excess containers will be returned.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

numPendingContainerRequests);

if (numPendingContainerRequests > 0) {
numPendingContainerRequests = Math.max(0, numPendingContainerRequests - 1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it enough to write numPendingContainerRequests--?
If numPendingContainerRequests is 0, it won't go into this branch in the next iteration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true

Copy link
Member

@GJL GJL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok.

}
}

// if we are waiting for no further containers, we can go to the
// regular heartbeat interval
if (numPendingContainerRequests <= 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think going negative should not be possible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes indeed. However, the logic should not be affected by using <= as the comparison operator.

Let all Yarn entry points use the YarnConfigOptions.APPLICATION_MASTER_PORT option
to specify the valid port range for the common RpcService.

This closes apache#5388.
…n result retrieval

Split RestClusterClient#submitJob into submitJob and requestJobResult which can
be called individually.

This closes apache#5428.
Register the JobTerminationHandler at the WebMonitorEndpoint to make it accessible to
all REST endpoints.

This closes apache#5429.
In order to support the job cancellation from the web UI, including when using
Yarn, we have to register the JobTerminationHandler under /jobs/:jobid/yarn-cancel
and /jobs/:jobid/yarn-stop. This is just a temporary fix until we can send
arbitrary REST verbs through the Yarn proxy.

This closes apache#5430.
The MiniDispatcher is responsible for submitting the single job with which
a job mode cluster is started. Once the job has completed and if the cluster
has been started in detached mode, the MiniDispatcher will terminate.

In order to reduce code duplication, the MiniDispatcher is a sub class of the
Dispatcher which is started with a single job submitted job graph store.

This closes apache#5431.
This commit allows to deploy detached job mode clusters via the
CliFrontend. In order to do that, it first extracts the JobGraph
from the PackagedProgram and then uses the ClusterDescriptor to
deploy the job mode cluster.

This closes apache#5432.
Upon notification of newly allocated containers, the YarnResourceManager
will only accept as many containers as there are pending container requests.
All excess containers will be returned.

This closes apache#5436.
@tillrohrmann
Copy link
Contributor Author

Thanks for the review @GJL. I addressed your comments. Merging this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants