New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
YARN-11533. CapacityScheduler CapacityConfigType changed in legacy queue allocation mode #5852
Conversation
…eue allocation mode
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM.
The root cause of this behavioural change is that from now on the root queue's capacity cannot be configured. Because, the configuredCapacityVector is set to PERCENTAGE mode (100%), regardless of the configuration. Later this is utilised in theRootCalculationDriver
.
public QueueCapacityVector parse(String capacityString, String queuePath) {
if (queuePath.equals(CapacitySchedulerConfiguration.ROOT)) {
return QueueCapacityVector.of(100f, ResourceUnitCapacityType.PERCENTAGE);
}
I think when the root.capacity is not configured, the 100% in percentage mode is a sane default. I'm not entirely sure what use cases would utilise a root.capacity other than 100% (maybe for a debug session to explicitly under-utilise a cluster, or being clever with node-labels).
Fixing this properly would require to change the RootCalculationDriver to handle the mixed capacityVector. If the root.capacity is not configured then we could always fallback to 100% or based on the root's children we could set either 1w, 100% or sum of children's absolute capacity if all the children has the same and only one capacity type, the latter is more cumbersome.
Currently, in legacy mode the root's absolute capacity most be greater to or equal to sum of the children. In percentage mode setting smaller than 100% have no effect. In weight it would not matter either what's the weight of the root queue.
Based on this, I think it should be OK to document this as is for the non-legacy queue mode (the root.capacity is always in percentage mode: 100% and cannot be configured), until there is a need to implement the mixed capacity handling for the root queue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Thanks for the reviews @tomicooler @sunilgovind! Merging to trunk. |
…eue allocation mode (apache#5852) Co-authored-by: Benjamin Teke <bteke@cloudera.com>
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?