[FEATURE] Full or partial Quota usage prevents Live Migration of VM when additional VM exceeds Quota

[rebeccazzzz]

Describe the bug

Fully or partially utilizing the Quota set on the Project and/or Namespace prevents the Live Migration of VMs.

The example below illustrates the problem and prevents the Live Migration of a VM.

• Quota: 64GB Memory
• VMs: 3 x 16GB Memory Nodes (together 48GB)

The expectation is that Live Migration could be done, but unfortunately the below message was seen.

///

(combined from similar events): Error creating pod: pods "virt-launcher-awhs-int-pool1-0c06dbc6-c7wnq-7rdpv" is forbidden: exceeded quota: default-gmp6p, requested: limits.memory=17405661185, used: limits.memory=52216983555, limited: limits.memory=64Gi

///
 
Also, even when a user fully utilizes its quota, the expectation is that the existing VMs are allowed to live migrate to another node.

Expected behavior

• Actual behavior:
 Live migration not possible when quota is set
• Expected behavior:
 Live migration should be possible with partial or full quota utilization

[chrisho]

After creating VMs, because the virt-component in the Pod requires a certain memory overhead, resulting in a higher actual quota of Pod memory than expected from the VM configuration, however, there is a possibility that the number of VMs created by the user or the number of simultaneous migrations will not reach the expectation.

Example:
The user configures the ResourceQuota of 10c and 10Gi, and the maintenance quota is 50% each, i.e. VM Available Resource and Maintenance Available Resource are 5c and 5Gi each. Although the user can create 5 VMs at this time, the 5 VMs cannot be migrated at the same time because the total memory used by the 5 VMs is more than 5Gi, so only 4 VMs can be migrated at the same time.
In general, Maintenance Resources will only be overused in accordance with general usage.

solution: https://github.com/chrisho/harvester/blob/maintenance-resource-hep/enhancements/20230228-resource-quota-enhancement.md#vm-overhead-resource

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
  The HEP PR is at: HEP for adjust the scale of ResourceQuota during migration [CI SKIP] #3814

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: [FEATURE] Full or partial Quota usage prevents Live Migration of VM when additional VM exceeds Quota #3124 (comment)

* [ ] Is there a workaround for the issue? If so, where is it documented?
The workaround is at:

* [ ] Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
The PR is at:

* [ ] Does the PR include the explanation for the fix or the feature?

* [ ] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
The PR for the YAML change is at:
The PR for the chart change is at:

* [ ] If labeled: area/ui Has the UI issue filed or ready to be merged?
The UI issue/PR is at:

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at: Add ResourceQuota doc docs#346

* [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?
- The automation skeleton PR is at:
- The automation test case PR is at:

* [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#741

[lanfon72]

Verified this feature has been implemented.

Test Information

• Environment: Baremetal DL360 3 nodes
• Harvester Version: v1.2.0-rc1
• ui-source Option: Auto
• Rancher Version:
  • v2.6-a4b924d21cd5d90b28ab8009edc18d96542952ac-head, Docker Image: a1d1e43988d2
  • v2.7-fbca7c34e5aeae36b85d8b0e9af12a2f6f13e0ea-head, Docker Image: 73f3b3456e62

Verify Steps

1. Install Harvester with at least 2 nodes
2. Import Harvester into Rancher
3. Access Harvester via Rancher's Virtualization Management
4. Navigate to Advanced/Settings, update overcommit-config to cpu:100, memory:100
5. Navigate to Projects/Namespaces, Create Project proj-a with Resource Quotas
   • CPU Limit: 5000 mCPUs/Project Limit, 3000 mCPUs/Namespace Default Limit
   • Memory Limit: 5000 MiB/Project Limit, 4000 mCPUs/Namespace Default Limit
6. Create Namespace ns-1 under Project proj-a
7. Create a image for VM creation
8. Create 2 VMs belong to namespace ns-1
   • Named vm1 with CPU: 1C, Memory 1GiB
   • Named vm2 with CPU: 1C, Memory 2GiB
9. Migrate vm1 and vm2 to another host concurrently
10. VMs should be migrated successfully