https://blueprints.launchpad.net/nova/+spec/use-pcpu-and-vcpu-in-one-instance
The spec CPU resource tracking splits host CPUs into PCPU
and VCPU
resources, making it possible to run instances of dedicated
CPU allocation policy and instances of shared
CPU allocation policy in the same host. This spec aims to create such kind of instance that some of the vCPUs are dedicated (PCPU
) CPUs and the remaining vCPUs are shared (VCPU
) vCPUs and expose this information via the metadata API.
The current CPU allocation policy, dedicated
or shared
, is applied to all vCPUs of an instance. However, with the introduction of CPU resource tracking, it is possible to propose a more fine-grained CPU allocation policy, which is based on the control over individual instance vCPU, and specifying the dedicated
or shared
CPU allocation policy to each instance vCPU.
As an operator, I would like to have an instance with some realtime CPUs for high performance, and at the same time, in order to increase instance density, I wish to make the remaining CPUs, which do not demand high performance, shared with other instances because I only care about the performance of realtime CPUs. One example is deploying the NFV task that is enhanced with DPDK framework in the instance, in which the data plane threads could be processed with the realtime CPUs and the control-plane tasks are scheduled on CPUs that may be shared with other instances.
As a Kubernetes administrator, I wish to run a multi-tier or auto-scaling application in Kubernetes, which is running in single OpenStack VM, with the expectation that using dedicated high-performance CPUs for application itself and deploying the containers on shared cores.
dedicated
and shared
are the existing instance CPU allocation policies that determine how instance CPU is scheduled on host CPU. This specification proposes a new CPU allocation policy, with the name mixed
, to create a CPU mixed instance in such way that some instance vCPUs are allocated from computing node's PCPU
resource, and the rest of instance vCPUs are allocated from the VCPU
resources. The CPU allocated from PCPU
resource will be pinned on particular host CPUs which are defined in CONF.compute.dedicated_cpu_set
, and the CPU from VCPU
resource will be floating on the host CPUs which are defined in CONF.compute.shared_cpu_set
. In this proposal, we call these two kinds of vCPUs as dedicated vCPU and shared vCPU respectively.
Nova operator may set the instance CPU allocation policy through the hw:cpu_policy
and hw_cpu_policy
interfaces, which may raise conflict. The CPU policy conflict is proposed to be solved with the following policy matrix:
+---------------------------+-----------+-----------+-----------+-----------+ | | hw:cpu_policy | + INSTANCE CPU POLICY +-----------+-----------+-----------+-----------+ | | DEDICATED | MIXED | SHARED | undefined | +---------------+-----------+-----------+-----------+-----------+-----------+ | hw_cpu_policy | DEDICATED | dedicated | conflict | conflict | dedicated | + +-----------+-----------+-----------+-----------+-----------+ | | MIXED | dedicated | mixed | conflict | mixed | + +-----------+-----------+-----------+-----------+-----------+ | | SHARED | dedicated | conflict | shared | shared | + +-----------+-----------+-----------+-----------+-----------+ | | undefined | dedicated | mixed | shared | undefined | +---------------+-----------+-----------+-----------+-----------+-----------+
For example, if a dedicated
CPU policy is specified in instance flavor hw:cpu_policy
, then the instance CPU policy is dedicated
, regardless of the setting specified in image property hw_cpu_policy
. If shared
is explicitly set in hw:cpu_policy
, then a mixed
policy specified in hw_cpu_policy
is conflict, which will throw an exception, the instance booting request will be rejected.
If there is no explicit instance CPU policy specified in flavor or image property, the flavor matrix result would be 'undefined', and the final instance policy is further determined and resolved by resources:PCPU
and resources:VCPU
specified in flavor extra specs. Refer to section <mixed-instance-PCPU-VCPU>
and the spec CPU resource tracking.
Real-time vCPU also occupies the host CPU exclusively and does not share CPU with other instances, all real-time vCPUs are dedicated vCPUs. For a mixed instance with some real-time vCPUs, with this proposal, the vCPUs not in the instance real-time vCPU list are shared vCPUs.
If emulator thread policy is ISOLATE
, the mixed instance will look for a dedicated host CPU for instance emulator thread, which is very similar to the case introduced by dedicated
policy instance.
If the emulator thread policy is SHARE
, then the instance emulator thread will float over the host CPUs defined in configuration CONF.compute.cpu_shared_set
.
As an interface to create the mixed
policy instance through legacy flavor extra specs or image properties, the flavor extra spec hw:cpu_dedicated_mask
is introduced. If the extra spec hw:cpu_dedicated_mask
is found in the instance flavor, then the information of the dedicated CPU could be found through parsing hw:cpu_dedicated_mask
.
Here is the example to create an instance with mixed
policy:
$ openstack flavor set <flavor_id> \
--property hw:cpu_policy=mixed \
--property hw:cpu_dedicated_mask=0-3,7
And, following is the proposing command to create a mixed instance which consists of multiple NUMA nodes by setting the dedicated vCPUs in hw:cpu_dedicated_mask
:
$ openstack flavor set <flavor_id> \
--property hw:cpu_policy=mixed \
--property hw:cpu_dedicated_mask=2,7 \
--property hw:numa_nodes=2 \
--property hw:numa_cpus.0=0-2 \
--property hw:numa_cpus.1=3-7 \
--property hw:numa_mem.0=1024 \
--property hw:numa_mem.1=2048
Note
Please be aware that there is no equivalent setting in image properties for flavor extra spec hw:cpu_dedicated_mask
. It will not be supported to create mixed instance through image properties.
Note
The dedicated vCPU list of a mixed instance could be specified through the newly introduced dedicated CPU mask or the cpu-time CPU mask, the hw:cpu_realtime_mask
or hw_cpu_realtime_mask
, you cannot set it by setting dedicated CPU mask extra spec and real-time CPU mask at the same time.
CPU resource tracking introduced a way to create an instance with dedicated
or shared
CPU allocation policy through resources:PCPU
and resources:VCPU
interfaces, but did not allow requesting both PCPU
resource and VCPU
resource for one instance.
This specification proposes to let an instance request PCPU
resource along with VCPU
, and effectively applying for the mixed
CPU allocation policy if the cpu_policy
is not explicitly specified in the flavor list. So an instance with such flavors potentially creates a mixed
policy instance:
$ openstack flavor set \
--property "resources:PCPU"="<dedicated CPU number>" \
--property "resources:VCPU"="<shared CPU number>" \
<flavor_id>
For mixed instance created in such way, both <shared CPU number> and <dedicated CPU number> must be greater than zero. Otherwise, it effectively creates the dedicated
or shared
policy instance, that all vCPUs in the instance is in a same allocation policy.
The resources:PCPU
and resources::VCPU
interfaces only put the request toward Placement
service for how many PCPU
and VCPU
resources are required to fulfill the instance vCPU thread and emulator thread requirement. The PCPU
and VCPU
distribution on the instance, especially on the instance with multiple NUMA nodes, will be spread across the NUMA nodes in the round-robin way, and VCPU
will be put ahead of PCPU
. Here is one example and the instance is created with flavor below:
flavor:
vcpus:8
memory_mb=512
extra_specs:
hw:numa_nodes:2
resources:VCPU=3
resources:PCPU=5
Instance emulator thread policy is not specified in the flavor, so it does not occupy any dedicated PCPU
resource for it, all PCPU
and VCPU
resources will be used by vCPU threads, and the expected distribution on NUMA nodes is:
NUMA node 0: VCPU VCPU PCPU PCPU
NUMA node 1: VCPU PCPU PCPU PCPU
Note
The demanding instance CPU number is the number of vCPU, specified by flavor.vcpus
, plus the number of CPU that is special for emulator thread, and if the emulator thread policy is ISOLATE
, the instance requests flavor.vcpus
+ 1 vCPUs, if the policy is not ISOLATE
, the instance just requests flavor.vcpus
vCPU.
Instead of adding a special instance CPU allocation policy, the CPU mixed instance is supported by extending the existing dedicated
policy and specifying the vCPUs that are pinned to the host CPUs chosen from PCPU
resource.
Following extra spec and the image property are defined to keep the dedicated vCPUs of a mixed
policy instance:
hw:cpu_dedicated_mask=<cpu set string>
hw_cpu_dedicated_mask=<cpu set string>
The <cpu set string>
shares the same definition defined above.
This was rejected at it overloads the dedicated
policy to mean two things, depending on the value of another configuration option.
Following commands was proposed as an example to create a mixed instance by an explicit request of PCPU
resources, and infer the VCPU
count by flavor::vcpus
and PCPU
count:
$ openstack flavor create mixed_vmf --vcpus 4 --ram 512 --disk 1
$ openstack flavor set mixed_vmf \
--property hw:cpu_policy=mixed \
--property resources:PCPU=2
This was rejected due to the mixing use of hw:cpu_policy
and resources:PCPU
. It is not recommended to mix placement style syntax with traditional extra specs.
Add the pcpuset
field in InstanceNUMACell
object to track the dedicated vCPUs of the instance NUMA cell, and the original InstanceNUMACell.cpuset
is special for shared vCPU then.
This change will introduce some database migration work for the existing instance in a dedicated
CPU allocation policy, since all vCPUs in such an instance are dedicated vCPUs which should be kept in pcpuset
field, but they are stored in cpuset
historically.
The metadata API will be extended with the dedicated vCPU info and a new OpenStack metadata version will be added to indicate this is a new metadata API.
The new field will be added to the meta_data.json
:
dedicated_cpus=<cpu set string>
The <cpu set string>
lists the dedicated vCPU set of the instance, which might be the content of hw:cpu_dedicated_mask
or hw:cpu_realtime_mask
or hw_cpu_realtime_mask
or the CPU list generated with the round-robin policy as described in section <mixed-instance-PCPU-VCPU>
.
The new cpu policy mixed
is added to extra spec hw:cpu_policy
.
None
None
If the end user wants to create an instance with a mixed
CPU allocation policy, the user is required to set corresponding flavor extra specs or image properties.
This proposal affects the selection of instance CPU allocation policy, but the performance impact is trivial.
None
None
The mixed
cpu policy is only available when the whole cluster upgrade finished. A service version will be bumped for detecting the upgrade.
The InstanceNUMACell.pcpuset
is introduced for dedicated vCPUs and the InstanceNUMACell.cpuset
is special for shared vCPUs, all existing instances in a dedicated
CPU allocation policy should be updated by moving content in InstanceNUMACell.cpuset
filed to InstanceNUMACell.pcpuset
field. The underlying database keeping the InstanceNUACell
object also need be updated to reflect this change.
- Primary assignee:
Wang, Huaqiang <huaqiang.wang@intel.com>
- Feature liaison:
Stephen Finucane <stephenfin@redhat.com>
- Add a new field, the
pcpuset
, forInstanceNUMACell
for dedicated vCPUs. - Add new instance CPU allocation policy
mixed
property and resolve conflicts - Bump nova service version to indicate the new CPU policy in nova-compute
- Add flavor extra spec
hw:cpu_dedicated_mask
and create mixed instance - Translate dedicated and shared CPU request to placement
PCPU
andVCPU
resources request. - Change libvirt driver to create
PCPU
mapping andVCPU
mapping - Add nova metadata service by offering final pCPU layout in
dedicated_cpus
field - Validate real-time CPU mask for
mixed
instance.
None
Functional and unit tests are required to cover:
- Ensure to solve the conflicts between the CPU policy matrix
- Ensure only dedicated vCPUs are possible to be real-time vCPUs
- Ensure creating
mixed
policy instance properly either by flavor settings or byresources::PCPU=xx
andresources::VCPU=xx
settings. - Ensure shared vCPUs is placed before the
dedicated
vCPUs - Ensure the emulator CPU is properly scheduled according to its policy.
The documents should be changed to introduce the usage of new mixed
CPU allocation policy and the new flavor extra specs.
Metadata service will be updated accordingly.
RevisionsRelease Name | Description |
---|---|
Train | Introduced, abandoned |
Ussuri | Approved |
Victoria | Re-proposed |