Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. #6150

HaiyangDING · 2015-03-30T04:54:05Z

This PR implements issue #5783.

Add a new priority function: BalancedResourceAllocation
BalancedResourceAllocation(BRA) rates a host based ONLY on how balanced its different types of resource are allocated if the pod would be scheduled on that host (metric: requested resource over capacity in terms of percentage for cpu and memory) . Score scales from 0 to 10 and 10 means the two percentage metrics are well very close, i.e. well balanced.
BRA PriorityFunction should NEVER be used alone and MUST be used collaboratively with LeastRequestedPriority function.
In some research articles, this method is referred to as Balanced Resource Utilization, and we choose to call it Balanced Resource Allocation for better readability.

Why BRA

In the current PriorityFunction libraries, the only function considering the resource utilization is the LeastRequestedPriority function which would lead to imbalanced resource allocation in some case. For instance, given two hosts with the same capacity {cpu: 10000, mem: 100000}, the current occupied resource is shown below:

host_name	cpu occupied	memory occupied
host1	3000	40000
host2	6000	10000

when a pod with {cpu: 2000, mem: 10000}, the LeastRequestedPriority would score the two hosts as:

host1:

cpuScore := (10000 - 3000 - 2000) / 10000 * 10 = 5
memScore := (100000 - 40000 - 10000) / 100000 * 10 = 5
score := (5 + 5) / 2 = 5

host2:

cpuScore := (10000 - 6000 - 2000) / 10000 * 10 = 2
memScore := (100000 - 10000 - 10000) / 100000 * 10 = 8
score := (8 + 2) / 2 = 5

one can see that host1.score equals to host2.score, in which case the algorithm would select either host1 or host2 based on a random decision. However, it is clear that if host2 is chosen the resource usage rate would be imbalanced with 80% cpu occupied and only 20% memory used, leaving 80% as "resource fragment".

Balanced Resource Allocation algorithm is here to address such problem. The rational underlying the algorithm is, for each host, evaluating the usage percentage of the resources (cpu and memoy in our case) assuming the pending pod already scheduled on that host and rating the host based on how balanced the concerned resources are used. That is to say, host with similar cpu usage percentage and memory usage percentage would be preferred over those without.

As for the example above, host1 would be preferred over host2 as host1 would end up with a more balanced resource usage rate.

Details of BRA

The score given by BRA is scaled from 0-10 with 10 being preferred. This goes with other PriorityFunctions. Note that BRA implemented here considers only the percentage of requested resource over capacity without taking into account the actual free resources on the host. Thus, BRA PriorityFunction should NEVER be used alone and MUST be used collaboratively with LeastRequestedPriority function.

Here are the steps showing the score is calculated:

step1: calculate the cpuFraction and memoryFraction

            fractionCapacity:= Requested / Capacity * 100%

step2: calculate the difference

            diff := abs(cpuFraction - memoryFraction)

step3: scale the difference to 0~10 and subtract by 10 to get the score

            score := 10 - diff*10

PS

PS1: If requested >= capacity, score is set to 0 directly so that host with such high resource usage would not be preferred.
PS3: For the example mentioned above, the final score would be:
```
host1.score = 10
host2.score = 4
```
so host1 is preferred by BRA function.

googlebot · 2015-03-30T04:54:07Z

Thanks for your pull request.

It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA) at https://cla.developers.google.com/.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check the information on your CLA or see this help article on setting the email on your git commits.

Once you've done that, please reply here to let us know. If you signed the CLA as a corporation, please let us know the company's name.

googlebot · 2015-03-30T05:55:44Z

CLAs look good, thanks!

erictune · 2015-03-30T16:14:28Z

I'd rather we don't use the word "Usage" for "Requested divided by Capacity", unless there is some existing standard for naming these things, which I'm unaware of.

To me, "Usage" is an absolute quantity, not a ratio. And, without qualifiers, it is a measure made by the operating system, not kubernetes pod requests. See for example, the Linux kernel cpuacct.usage cgroup file.

Call this PctOfCapRequested or something.

erictune · 2015-03-30T16:17:08Z

By the same logic, please don't call this Utilization. IIUC, when people talk about utilization in a datacenter, without any qualifiers, they mean usage divided by capacity, not request divided by capacity.

Maybe call this Balanced Node Fullness, or something.

HaiyangDING · 2015-03-31T06:45:30Z

Hi, @erictune

Thank you for your comments.

I agree that using the word "Usage" for "Requested divided by Capacity" is not proper, and they will be replaced by "PctOfCapRequested" in both code and the PR message.

Regarding the word "Utilization" in Balanced Resource Utilization, this word is actually used in 2 research articles, i.e.

Xin Li et al, Balancing Resource Utilization for Continuous Virtual Machine Requests in Clouds, Sixth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, 2012
Wei Huang et al, An Energy Efficient Virtual Machine Placement Algorithm with Balanced Resource Utilization, Seventh International Conference on Innovative Mobile and Internet Services in Ubiquitous Computin, 2013

However, if this word is confusing in this scenario, I plan to change it to Balanced Resource Allocation. What do you think?

HaiyangDING · 2015-04-01T02:10:28Z

Need some time, trying to fix check failure ASAP.

bgrant0607 · 2015-04-01T06:06:49Z

cc @davidopp @abhgupta

HaiyangDING · 2015-04-01T07:19:30Z

Hi, folks

My last commit has failed the continous-integration check and the reason is (as given by the details from https://travis-ci.org/GoogleCloudPlatform/kubernetes/jobs/56587523):

FAIL    github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider/controller  0.280s

...

!!! Error in ./hack/test-go.sh:168

'xargs -I{} -n1 -P${KUBE_COVERPROCS} go test "${goflags[@]:+${goflags[@]}}" ${KUBE_RACE} ${KUBE_TIMEOUT} -cover -covermode="${KUBE_COVERMODE}" -coverprofile="${cover_report_dir}/{}/${cover_profile}" "${cover_params[@]+${cover_params[@]}}" "${KUBE_GO_PACKAGE}/{}"' exited with status 0 123

Call stack:

1: ./hack/test-go.sh:168 runTests(...)
2: ./hack/test-go.sh:207 main(...)
Exiting with status 1

However, I can not reproduce this error in my local environment, i.e. ./hacktest-go.sh end up with all oks.

I am confused about this right now, so it would be great if someone could give any advice on this. Big thanks in advance!

abhgupta · 2015-04-01T16:26:18Z

@HaiyangDING Seems to be an unrelated error due to a timeout - I would just retry the test. One way to kick off the tests again is to just modify the commit.

bgrant0607 · 2015-04-01T17:09:47Z

I'll restart the tests.

erictune · 2015-04-01T18:15:20Z

@HaiyangDING
How about call it BalancedResourceAllocation, as you suggested, in the code and you can mention in the docstring that it is based on "Balanced Resource Utilization" research.

HaiyangDING · 2015-04-02T00:47:33Z

@erictune
Sounds like a great idea. Will implement sooner.

@abhgupta
I would try it next time, thank you.

@bgrant0607
Thank you for restarting the test.

HaiyangDING · 2015-04-03T00:59:53Z

@erictune

PTAL, I have fixed the code and the description.

davidopp · 2015-04-03T23:10:53Z

@erictune I assigned this to you just now. If you are too busy, I think @lavalamp said his PR review load is pretty light.

lavalamp · 2015-04-03T23:31:48Z

pkg/scheduler/priorities.go

+	capacityMilliCPU := node.Status.Capacity.Cpu().MilliValue()
+	capacityMemory := node.Status.Capacity.Memory().Value()
+
+	cpuPctOfCapRequested := calculatePctOfCapRequested(totalMilliCPU, capacityMilliCPU, node.Name)


suggest cpuFraction as the variable name. (and memoryFraction, averageFration below)

Sorry for disagreeing with previous reviewer :(

lavalamp · 2015-04-03T23:58:28Z

pkg/scheduler/priorities.go

+	} else {
+		score = int(1 / diff)
+	}
+	// if Requesetd/Capacity exceeds 100%, the PctOfCapRequested is set to 1 (100%) and the host should never be preferred.


Why not just directly check for request >= capacity above (before the call to calculatePctOfRequested) and return early if so?

lavalamp · 2015-04-04T00:04:00Z

Thanks for this change! I have a few comments. They're mostly minor.

Please squash your commits, in case I forget to ask later.

HaiyangDING · 2015-04-06T14:53:02Z

@lavalamp

Hi, thank you for the comments. I have just returned from a small vacation. Will be working on it sooner.

HaiyangDING · 2015-04-07T16:51:05Z

New patch submitted at my home where tests are unable to perform, I will fix them tomorrow. Sorry for the trouble.

bgrant0607 · 2015-04-07T17:46:24Z

pkg/scheduler/priorities_test.go

+		"baz": "blah",
+	}
+	machine1Status := api.PodStatus{
+		Host: "machine1",


Needs to be rebased. This field is gone.

Thank you, I will fix them.

HaiyangDING · 2015-04-08T16:29:13Z

How to deal with Shippable failures? I can not see the details.

lavalamp · 2015-04-08T21:44:08Z

Shippable is an unrelated flake. Last request before LGTM: document sources, and squash commits again. Thanks!

googlebot · 2015-04-09T07:33:40Z

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

HaiyangDING · 2015-04-09T09:15:04Z

I confirm that all contributors to this PR are OK with these commits being contributed to Google.

As a matter of fact, the commits are all made by me (dingh, HaiyangDING, dinghaiyang@huawei.com) but throught different PCs

PS: I will squash all the commits as I did in HaiyangDING@15b91cd later

HaiyangDING · 2015-04-09T09:16:37Z

I confirm that all contributors to this PR are OK with these commits being contributed to Google.

As a matter of fact, the commits are all made by me (dingh, HaiyangDING, dinghaiyang@huawei.commailto:dinghaiyang@huawei.com) but throught different PCs

丁海洋Haiyang DING,
华为技术有限公司.
Technology Introduction & Standard and Patent Dept,
Carrier Software Business Unit,
Huawei Technologies.
地址：南京市雨花区软件大道 101 号.
Address: Huawei Nanjing R&D Center
101 Software Avenue,Yuhuatai District, Nanjing 210012
Jiangsu, P.R.China
Tel: 0086-025-56620756
Email: dinghaiyang@huawei.commailto:huruifeng@huawei.com
http://www.huawei.com

发件人: googlebot [mailto:notifications@github.com]
发送时间: 2015年4月9日 15:34
收件人: GoogleCloudPlatform/kubernetes
抄送: dinghaiyang
主题: Re: [kubernetes] Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. (#6150)

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/6150#issuecomment-91137643.

HaiyangDING · 2015-04-09T10:56:14Z

@lavalamp

I think I have squashed all the commits, but it seems I have to do git pull + git push after git rebase -i which brings up the last commit (Merge branch 'bru' of https://github.com/HaiyangDING/kubernetes into bru 952c182). Details are at below.

I wonder if it is OK. Please bear me with this, I am new to squashing and let me know if anything should be done.

------------------- details of last steps of git rebase --------------

openstack@network:/dingh/kubernetes$ git rebase --continue
[detached HEAD 4507b9f] Implement BRA algorithm as a new priority function in scheduler
2 files changed, 309 insertions(+)
[detached HEAD 2bb7eeb] Implement BRA algorithm as a new priority function in scheduler
2 files changed, 309 insertions(+)
Successfully rebased and updated refs/heads/bru.
openstack@network:/dingh/kubernetes$ git push
Username for 'https://github.com': dinghaiyang@huawei.com
Password for 'https://dinghaiyang@huawei.com@github.com':
To https://github.com/HaiyangDING/kubernetes.git
! [rejected] bru -> bru (non-fast-forward)
error: failed to push some refs to 'https://github.com/HaiyangDING/kubernetes.git'
To prevent you from losing history, non-fast-forward updates were rejected
Merge the remote changes (e.g. 'git pull') before pushing again. See the
'Note about fast-forwards' section of 'git push --help' for details.
openstack@network:/dingh/kubernetes$ git pull
Merge made by the 'recursive' strategy.
openstack@network:/dingh/kubernetes$ git push
Username for 'https://github.com': dinghaiyang@huawei.com
Password for 'https://dinghaiyang@huawei.com@github.com':
To https://github.com/HaiyangDING/kubernetes.git
5387fab..952c182 bru -> bru
openstack@network:~/dingh/kubernetes$ git push --help

lavalamp · 2015-04-09T17:11:12Z

@HaiyangDING use fetch + rebase instead of pull + merge to prevent that.

googlebot · 2015-04-10T02:31:46Z

CLAs look good, thanks!

Balanced Resource Allocation policy can now be enabled to so that host(s) with balanced resource usage would be preferred. The score given by BRA also scales from 0 to 10 with 10 representing that the resource usage is well balanced.

HaiyangDING · 2015-04-10T03:06:50Z

@lavalamp

PTAL, finally it works... thanks

lavalamp · 2015-04-10T19:48:24Z

LGTM, thank you!

Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package.

googlebot added the cla: no label Mar 30, 2015

davidopp self-assigned this Mar 30, 2015

googlebot added cla: yes and removed cla: no labels Mar 30, 2015

HaiyangDING changed the title ~~Implement Balanced Resource Utilization (BRU) algorithm as a PriorityFunction in scheduler package.~~ Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. Apr 2, 2015

davidopp assigned erictune and unassigned davidopp Apr 3, 2015

davidopp added team/master priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Apr 3, 2015

erictune assigned lavalamp and unassigned erictune Apr 3, 2015

lavalamp reviewed Apr 3, 2015
View reviewed changes

bgrant0607 reviewed Apr 7, 2015
View reviewed changes

googlebot added cla: no and removed cla: yes labels Apr 9, 2015

HaiyangDING force-pushed the bru branch from cdf2e69 to 319f11e Compare April 10, 2015 02:31

googlebot added cla: yes and removed cla: no labels Apr 10, 2015

HaiyangDING force-pushed the bru branch from 319f11e to 71b2af1 Compare April 10, 2015 02:55

lavalamp added a commit that referenced this pull request Apr 10, 2015

Merge pull request #6150 from HaiyangDING/bru

2bd95d4

Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package.

lavalamp merged commit 2bd95d4 into kubernetes:master Apr 10, 2015

HaiyangDING deleted the bru branch July 1, 2015 09:44

HaiyangDING unassigned lavalamp Aug 12, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. #6150

Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. #6150

HaiyangDING commented Mar 30, 2015

googlebot commented Mar 30, 2015

googlebot commented Mar 30, 2015

erictune commented Mar 30, 2015

erictune commented Mar 30, 2015

HaiyangDING commented Mar 31, 2015

HaiyangDING commented Apr 1, 2015

bgrant0607 commented Apr 1, 2015

HaiyangDING commented Apr 1, 2015

abhgupta commented Apr 1, 2015

bgrant0607 commented Apr 1, 2015

erictune commented Apr 1, 2015

HaiyangDING commented Apr 2, 2015

HaiyangDING commented Apr 3, 2015

davidopp commented Apr 3, 2015

lavalamp Apr 3, 2015

lavalamp Apr 3, 2015

lavalamp Apr 3, 2015

lavalamp commented Apr 4, 2015

HaiyangDING commented Apr 6, 2015

HaiyangDING commented Apr 7, 2015

bgrant0607 Apr 7, 2015

HaiyangDING Apr 8, 2015

HaiyangDING commented Apr 8, 2015

lavalamp commented Apr 8, 2015

googlebot commented Apr 9, 2015

HaiyangDING commented Apr 9, 2015

HaiyangDING commented Apr 9, 2015

HaiyangDING commented Apr 9, 2015

lavalamp commented Apr 9, 2015

googlebot commented Apr 10, 2015

HaiyangDING commented Apr 10, 2015

lavalamp commented Apr 10, 2015

Navigation Menu

Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. #6150

Implement Balanced Resource Allocation (BRA) algorithm as a PriorityFunction in scheduler package. #6150

Conversation

HaiyangDING commented Mar 30, 2015

Why BRA

Details of BRA

PS

googlebot commented Mar 30, 2015

googlebot commented Mar 30, 2015

erictune commented Mar 30, 2015

erictune commented Mar 30, 2015

HaiyangDING commented Mar 31, 2015

HaiyangDING commented Apr 1, 2015

bgrant0607 commented Apr 1, 2015

HaiyangDING commented Apr 1, 2015

abhgupta commented Apr 1, 2015

bgrant0607 commented Apr 1, 2015

erictune commented Apr 1, 2015

HaiyangDING commented Apr 2, 2015

HaiyangDING commented Apr 3, 2015

davidopp commented Apr 3, 2015

lavalamp Apr 3, 2015

Choose a reason for hiding this comment

lavalamp Apr 3, 2015

Choose a reason for hiding this comment

lavalamp Apr 3, 2015

Choose a reason for hiding this comment

lavalamp commented Apr 4, 2015

HaiyangDING commented Apr 6, 2015

HaiyangDING commented Apr 7, 2015

bgrant0607 Apr 7, 2015

Choose a reason for hiding this comment

HaiyangDING Apr 8, 2015

Choose a reason for hiding this comment

HaiyangDING commented Apr 8, 2015

lavalamp commented Apr 8, 2015

googlebot commented Apr 9, 2015

HaiyangDING commented Apr 9, 2015

HaiyangDING commented Apr 9, 2015

HaiyangDING commented Apr 9, 2015

lavalamp commented Apr 9, 2015

googlebot commented Apr 10, 2015

HaiyangDING commented Apr 10, 2015

lavalamp commented Apr 10, 2015