YARN-11566. Yarn app kill command can not kill the application in sec… #6068
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
There are three ways to solve this problem:
Solution 1: Solution 2: Here I will choose solution 3. UAM is managed by NM, so this solution is most reasonable. This PR is about solution 3. |
Contributor
@zhengchenyu Thanks for your contribution! Can we describe what |
slfan1989
reviewed
Sep 14, 2023
| */ | ||
| @Override | ||
| public void shutdown() { | ||
| public void shutdown(boolean stop) { |
Contributor
There was a problem hiding this comment.
YARN-6848 plans to unify the Interceptor interface. Can we not change the shutdown parameters? Are there any other ways we can solve the issue?
slfan1989
reviewed
Sep 14, 2023
| if (response == null) { | ||
| throw new YarnException( | ||
| "Failed Force-killing UAM id " + uamId + " for application " + appId); | ||
| LOG.error("Failed Force-killing UAM id " + uamId + " for application " + appId); |
Contributor
There was a problem hiding this comment.
LOG.error("Failed Force-killing UAM id {} for application {}.", appId);
slfan1989
reviewed
Sep 14, 2023
| Assert.assertEquals(2, unmanagedAppMasterMap.size()); | ||
|
|
||
| // threadpool may be shutdown before finishApplicationThread is executed. | ||
| threadpool.shutdownNow(); |
Contributor
There was a problem hiding this comment.
In actual situations, what situations would cause the thread pool to be terminated?
slfan1989
reviewed
Sep 14, 2023
| Assert.assertEquals(2, interceptor.getUnmanagedAMPoolSize()); | ||
|
|
||
| // Allocate the second batch of containers, with sc1 and sc3 active | ||
| deRegisterSubCluster(SubClusterId.newInstance("SC-2")); |
Contributor
There was a problem hiding this comment.
In unit test, what is the purpose of deRegisterS SC-2 offline to verify?
Contributor
Author
I did not describe this clearly. It happen when the stop is true for shutdown. It will happen just after this PR. Firstly I will find a better way of shutdown for kill application. Then resolve other problem. |
|
💔 -1 overall
This message was automatically generated. |
…ondary sub cluster.
zhengchenyu
force-pushed
the
YARN-11566
branch
from
6325294 to
b995fa8
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
slfan1989
approved these changes
Sep 18, 2023
Contributor
|
Quick review, LGTM, read this pr carefully later. |
Contributor
|
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
When AMRMProxy is enable, the application may allocate container among multi sub cluster. The application in secondary sub cluster will be labeled as unmananged application. When we run 'yarn app -kill {appid}', the unmananged application will not be killed in secondary sub cluster.
The unmanaged application will be removed util app attempt is expired after 15 minute.
How was this patch tested?
unit test and test in real cluster.
For code changes:
interceptor.shutdown. So UnmanagedAMPoolManager::unmanagedAppMasterMap may be removed by interceptor.shutdown. And threadpool may be shutdown by interceptor.shutdown. So I use a copy of unmanagedAppMasterMap. And do not use threadpool. Because ForceFinishApplicationThread is a async thread, will not stuck serviceStop, to avoid to allocate new resource, here I just force kill application sequentially.