Fixed failed to start instances on Xenserver with dynamic scaling due to default max cpus. #4447

Spaceman1984 · 2020-11-05T08:54:54Z

Description

When trying to start an instance with dynamic scaling on a Xenserver Hypervisor with less than 16 cores, the VM fails to start. This is due to the default max value being set to 16. This PR limits the max vCPUs to what the host is capable of.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Set enable.dynamic.scale.vm to true
Try to start an instance on a Xenserver host with less than 16 cores.

Spaceman1984 · 2020-11-05T08:55:45Z

@blueorangutan package

blueorangutan · 2020-11-05T09:05:21Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-11-05T09:37:22Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2336

Spaceman1984 · 2020-11-06T03:47:02Z

@blueorangutan test centos7 xcpng76

blueorangutan · 2020-11-06T03:48:13Z

@Spaceman1984 a Trillian-Jenkins test job (centos7 mgmt + xcpng76) has been kicked to run smoke tests

harikrishna-patnala · 2020-11-06T03:52:03Z

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/XenServerGuru.java

-                to.setVcpuMaxLimit(MaxNumberOfVCPUSPerVM.valueIn(host.getClusterId()));
+                // Max cpu cannot be more than host capability
+                if (host.getCpus() < MaxNumberOfVCPUSPerVM.valueIn(host.getClusterId()))
+                    to.setVcpuMaxLimit(host.getCpus());


I think host.getCpus() get the actual CPU on the physical host, but xenserver host is capable of allocating virtual CPUs more than the actual count.
Here is the list of configuration limits for XS 7.1 version. https://docs.citrix.com/en-us/xenserver/7-1/system-requirements/configuration-limits.html
It says "Virtual CPUs per VM (Linux)" 32. So we might need to set the value according to this.

I tested this with a host with 8 CPUs and it failed trying to allocate 16 on XCP-ng 7.6

I'll do some testing on Xenserver

I tested XCP-NG 7.6 and Xenserver 7.6, I was unable to start a VM with more vCPUs than pCPUs.

Seems like overcommitting CPU on Xenserver was removed according to this article @harikrishna-patnala https://support.citrix.com/article/CTX236977

@Spaceman1984 got it, thanks for the article and yes it seems over committing CPU is disabled.
Do you think we have to also provide some global config "vcpu-unrestricted" to allow overcommit of CPU like xenserver does this way "xe vm-param-set uuid= platform:vcpu-unrestricted=true". This gives admins to choose whether to overcommit or not.

Otherwise If we have to go with by setting only actual pCPUs, we need to remove this configuration parameter "xen.vm.vcpu.max" as we dont need it now.

I kept the global var in to allow things like if you have a host with 32 pCPUs but want to set a lower max vCPU limit for a VM for the purpose of dynamic scaling.
As for the ability to force overcommit - This is not recommended in the article and I would think not best practice, that's why I would vote no.

please put a log message which explans why we are setting that value, otherwise user may think why a different value is set other than "xen.vm.vcpu.max"

Alright, awesome, on it.

yadvr · 2020-11-06T09:09:04Z

@blueorangutan package

yadvr · 2020-11-06T09:10:36Z

@blueorangutan package

blueorangutan · 2020-11-06T09:11:09Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-11-06T09:43:03Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2337

Spaceman1984 · 2020-11-06T10:12:42Z

@blueorangutan test centos7 xcpng76

blueorangutan · 2020-11-06T10:13:14Z

@Spaceman1984 a Trillian-Jenkins test job (centos7 mgmt + xcpng76) has been kicked to run smoke tests

blueorangutan · 2020-11-06T13:31:31Z

Trillian test result (tid-3130)
Environment: xcpng76 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33232 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4447-t3130-xcpng76.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	10.39	test_scale_vm.py

DaanHoogland · 2020-11-06T15:38:01Z

looking by the discussion here, we should move this to 4.16 and make a good functional definition of what is and isn't possible on different xen platforms.

blueorangutan · 2020-11-06T20:09:07Z

Trillian test result (tid-3131)
Environment: xcpng76 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33933 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4447-t3131-xcpng76.zip
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	9.32	test_scale_vm.py

DaanHoogland · 2020-11-10T20:54:42Z

looking by the discussion here, we should move this to 4.16 and make a good functional definition of what is and isn't possible on different xen platforms.

ping @Spaceman1984 @harikrishna-patnala . cc @rhtyd

Spaceman1984 · 2020-11-11T07:25:11Z

@DaanHoogland I have determined this PR is related to #4443, I will merge this branch into that one and then close this PR.

harikrishna-patnala · 2020-11-11T08:10:02Z

looking by the discussion here, we should move this to 4.16 and make a good functional definition of what is and isn't possible on different xen platforms.

ping @Spaceman1984 @harikrishna-patnala . cc @rhtyd

@DaanHoogland agreed to keep it for 4.16 since this makes a substantial change in the existing flow or values assigned.

Limiting max cpus to max host value

27ceb7c

Spaceman1984 closed this Nov 6, 2020

Spaceman1984 reopened this Nov 6, 2020

harikrishna-patnala reviewed Nov 6, 2020

View reviewed changes

Added log message

e0cc29e

yadvr requested review from DaanHoogland and yadvr November 6, 2020 09:09

yadvr added this to the 4.15.0.0 milestone Nov 6, 2020

Spaceman1984 added the Severity:Major label Nov 6, 2020

ACSGitBot added the type:bug label Nov 8, 2020

DaanHoogland added the component:XenServer label Nov 10, 2020

DaanHoogland modified the milestones: 4.15.0.0, 4.16.0.0 Nov 11, 2020

Spaceman1984 closed this Nov 11, 2020

Spaceman1984 mentioned this pull request Nov 11, 2020

Fixed VM scaling test on Xenserver #4443

Closed

5 tasks

Fixed failed to start instances on Xenserver with dynamic scaling due to default max cpus. #4447

Fixed failed to start instances on Xenserver with dynamic scaling due to default max cpus. #4447

Uh oh!

Conversation

Spaceman1984 commented Nov 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Spaceman1984 commented Nov 5, 2020

Uh oh!

blueorangutan commented Nov 5, 2020

Uh oh!

blueorangutan commented Nov 5, 2020

Uh oh!

Spaceman1984 commented Nov 6, 2020

Uh oh!

blueorangutan commented Nov 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Spaceman1984 Nov 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Spaceman1984 Nov 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yadvr commented Nov 6, 2020

Uh oh!

yadvr commented Nov 6, 2020

Uh oh!

blueorangutan commented Nov 6, 2020

Uh oh!

blueorangutan commented Nov 6, 2020

Uh oh!

Spaceman1984 commented Nov 6, 2020

Uh oh!

blueorangutan commented Nov 6, 2020

Uh oh!

blueorangutan commented Nov 6, 2020

Uh oh!

DaanHoogland commented Nov 6, 2020

Uh oh!

blueorangutan commented Nov 6, 2020

Uh oh!

DaanHoogland commented Nov 10, 2020

Uh oh!

Spaceman1984 commented Nov 11, 2020

Uh oh!

harikrishna-patnala commented Nov 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Spaceman1984 commented Nov 5, 2020 •

edited

Loading

Spaceman1984 Nov 6, 2020 •

edited

Loading

Spaceman1984 Nov 6, 2020 •

edited

Loading