Skip to content

[Feature] Add AccessManager #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
duanmeng merged 5 commits into from
Jan 19, 2022
Merged

[Feature] Add AccessManager #61

duanmeng merged 5 commits into from
Jan 19, 2022

Conversation

@duanmeng
Copy link
Collaborator

@duanmeng duanmeng commented Jan 5, 2022

What changes were proposed in this pull request?

Add access manager to handle cron task access requests and add checkers to check the candidate list and cluster load respectively.

Why are the changes needed?

We need to add candidates checking and control of the access when the cluster load is high.

Does this PR introduce any user-facing change?

Yes, need to update the client to enable this feature.

How was this patch tested?

UT and system test.

@duanmeng duanmeng added the enhancement New feature or request label Jan 5, 2022
jerqi
jerqi reviewed Jan 5, 2022
View reviewed changes
@frankliee frankliee changed the title Add AccessManager [Feature] Add AccessManager Jan 5, 2022
colinmjj
colinmjj reviewed Jan 5, 2022
View reviewed changes
frankliee
frankliee reviewed Jan 5, 2022
View reviewed changes
jerqi
jerqi reviewed Jan 7, 2022
View reviewed changes
colinmjj
colinmjj reviewed Jan 10, 2022
View reviewed changes
@duanmeng duanmeng mentioned this pull request Jan 11, 2022
Merged
@duanmeng duanmeng force-pushed the addAccessManager branch 2 times, most recently from 2b09a38 to 8b0c874 Compare January 12, 2022 06:42
@duanmeng duanmeng force-pushed the addAccessManager branch 2 times, most recently from 83b0fc8 to 4180592 Compare January 12, 2022 08:38
colinmjj
colinmjj reviewed Jan 12, 2022
View reviewed changes
@duanmeng duanmeng force-pushed the addAccessManager branch 3 times, most recently from f20ca88 to 65818c1 Compare January 12, 2022 11:14
colinmjj
colinmjj previously approved these changes Jan 13, 2022
View reviewed changes
Copy link
Collaborator

@colinmjj colinmjj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

colinmjj
colinmjj previously approved these changes Jan 13, 2022
View reviewed changes
jerqi
jerqi reviewed Jan 13, 2022
View reviewed changes
@duanmeng duanmeng force-pushed the addAccessManager branch 4 times, most recently from f0c70cf to 2d67fec Compare January 13, 2022 08:02
jerqi
jerqi reviewed Jan 14, 2022
View reviewed changes
@duanmeng
Copy link
Collaborator Author

duanmeng commented Jan 17, 2022

@colinmjj @jerqi @frankliee all comments resolved

@duanmeng duanmeng merged commit 2cd2175 into Tencent:master Jan 19, 2022
duanmeng added a commit that referenced this pull request Jan 26, 2022
@duanmeng
[Feature]Access check in client (#66)
ece77d1 
# What changes were proposed in this pull request?
Add a delegation shuffle manager
choose the actual shuffle manager (sort or rss)
# Why are the changes needed?
This pr is related to #61, which add access client.

# Does this PR introduce any user-facing change?
Yes, client need to config spark.shuffle.manager explicitly and add the following config , spark.rss.accessInfoKey it specifies which is the config key of the accessInfo and then add the config duration submition.

# How was this patch tested?
UT and system test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@frankliee frankliee frankliee left review comments

@jerqi jerqi jerqi approved these changes

@colinmjj colinmjj colinmjj left review comments

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@duanmeng @colinmjj @jerqi @frankliee