YARN-9834. Allow using a pool of local users to run Yarn Secure Conta… #1446
+617
−7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…iner in secure mode Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
|
💔 -1 overall
This message was automatically generated. |
Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
| LOG.error(errMsg); | ||
| return; | ||
| } | ||
|
|
| localUserInfo.fileOpCount--; | ||
| LOG.info("Decremented fileOpCount for appUser " + appUser + | ||
| " to " + localUserInfo.fileOpCount); | ||
|
|
| } | ||
| if (index == -1) { | ||
| String errMsg = "Cannot allocate local users from a pool of " + | ||
| localUserCount; |
| int appCount; | ||
| int fileOpCount; | ||
| int logHandlingCount; | ||
|
|
| * Maintains the appUser to local user mapping, until: | ||
| * a) all applications of the appUser is finished; | ||
| * b) all FileDeletionTask for that appUser is executed; | ||
| * c) all log aggregation/handling requests for appUser's applications are done |
|
💔 -1 overall
This message was automatically generated. |
Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
| } | ||
| if (index == -1) { | ||
| String errMsg = "Cannot allocate local users from a pool of " + | ||
| localUserCount; |
| int appCount; | ||
| int fileOpCount; | ||
| int logHandlingCount; | ||
|
|
|
💔 -1 overall
This message was automatically generated. |
Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
| } | ||
| if (index == -1) { | ||
| String errMsg = "Cannot allocate local users from a pool of " + | ||
| localUserCount; |
| int appCount; | ||
| int fileOpCount; | ||
| int logHandlingCount; | ||
|
|
|
💔 -1 overall
This message was automatically generated. |
ashvina
suggested changes
Sep 17, 2019
| @@ -1611,7 +1625,7 @@ private void cleanUpFilesPerUserDir(FileContext lfs, DeletionService del, | |||
| String owner = status.getOwner(); | |||
| List<Path> pathList = new ArrayList<>(); | |||
| pathList.add(status.getPath()); | |||
| FileDeletionTask deletionTask = new FileDeletionTask(del, owner, null, | |||
| FileDeletionTask deletionTask = new FileDeletionTask(del, null, null, | |||
There was a problem hiding this comment.
Is this change intentional? I am not sure if I understand why the information of the user deleting the file is not required?
| YarnConfiguration.NM_SECURE_MODE_USE_POOL_USER, | ||
| YarnConfiguration.DEFAULT_NM_SECURE_MODE_USE_POOL_USER); | ||
| if (this.disablePrivateVis) { | ||
| LOG.info("When " + YarnConfiguration.NM_SECURE_MODE_USE_POOL_USER + |
There was a problem hiding this comment.
All resources can either be public or application. It seems this constraint is added since a local-user may get assigned to different real-users. Can you comment on the cases where a real user application may require private resources and local-user-pooling is enabled?
There was a problem hiding this comment.
Sure, will add comments. Basically if a real user application require private resources, when local user pooling is enabled, we will treat the private resources as application resources. This means for each application we will download (localize) the resources to application folder, and will delete it after the application is finished.
…ainer Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
| } | ||
| if (index == -1) { | ||
| String errMsg = "Cannot allocate local users from a pool of " + | ||
| localUserCount; |
| int appCount; | ||
| int fileOpCount; | ||
| int logHandlingCount; | ||
|
|
|
💔 -1 overall
This message was automatically generated. |
…ainer Fix the issue when deallocating local user for an appuser, usercache/<appuser>/appcache folder still exists. After deallocating appUser from localUser, we will check if usercache folder for the appUser exists, if yes, then queue FileDeletionTasks to delete. Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
hadoop-yetus
reviewed
Oct 5, 2019
| } | ||
| if (index == -1) { | ||
| String errMsg = "Cannot allocate local users from a pool of " + | ||
| localUserCount; |
hadoop-yetus
reviewed
Oct 5, 2019
| int appCount; | ||
| int fileOpCount; | ||
| int logHandlingCount; | ||
|
|
|
💔 -1 overall
This message was automatically generated. |
Signed-off-by: Shanyu Zhao <shzhao@microsoft.com>
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…iner in secure mode
Signed-off-by: Shanyu Zhao shzhao@microsoft.com
https://issues.apache.org/jira/browse/YARN-9834