Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge "libvirt: virtio-net multiqueue"
- Loading branch information
Showing
1 changed file
with
196 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,196 @@ | ||
.. | ||
This work is licensed under a Creative Commons Attribution 3.0 Unported | ||
License. | ||
http://creativecommons.org/licenses/by/3.0/legalcode | ||
|
||
========================================== | ||
libvirt: virtio-net multiqueue | ||
========================================== | ||
|
||
https://blueprints.launchpad.net/nova/+spec/libvirt-virtio-net-multiqueue | ||
|
||
This blueprint aims to enhance the libvirt driver to enable the virtio-net | ||
multiqueue. This will allow the guest instances to increase the total network | ||
throughput. | ||
|
||
Problem description | ||
=================== | ||
|
||
Today's high-end servers have more processors, and guests running on them often | ||
have an increasing number of vCPUs. In a single virtio-net queue, the scale of | ||
the protocol stack in a guest is restricted, as the network performance does | ||
not scale as the number of vCPUs increases. Guests cannot transmit or | ||
retrieve packets in parallel, as virtio-net has only one TX and RX queue. | ||
|
||
Multiqueue virtio-net provides the greatest performance benefit when: | ||
|
||
- Traffic packets are relatively large. | ||
- The guest is active on many connections at the same time, with traffic | ||
running between guests, guest to host, or guest to an external system. | ||
- The number of queues is equal to the number of vCPUs. This is because | ||
multi-queue support optimizes RX interrupt affinity and TX queue | ||
selection in order to make a specific queue private to a specific vCPU. | ||
|
||
Although the virtio-net multiqueue feature provides a performance benefit, | ||
it has some limitations and therefore should not be unconditionally enabled: | ||
|
||
- Guest OS is limited to ~200 MSI vectors. Each NIC queue requires a MSI | ||
vector, as well as any virtio device or assigned PCI device. | ||
Defining an instance with multiple virtio NICs and vCPUs might lead to a | ||
possibility of hitting the guest MSI limit. | ||
- virtio-net multiqueue works well for incoming traffic, but can | ||
occasionally cause a performance degradation, for outgoing traffic. | ||
- Enabling virtio-net multiqueue increases the total network throughput, | ||
but in parallel it also increases the CPU consumption. | ||
- Enabling virtio-net multiqueue in the host QEMU config, does not enable | ||
the functionality in the guest OS. The guest OS administrator needs to | ||
manually turn it on for each guest NIC that requires this feature, using | ||
ethtool. | ||
- MSI vectors would still be consumed (wasted), if multiqueue was enabled | ||
in the host, but has not been enabled in the guest OS by the administrator. | ||
- In case the number of vNICs in a guest instance is proportional to the | ||
number of vCPUs, enabling the multiqueue feature is less important. | ||
- Each virtio-net queue consumes 64 KB of kernel memory for the vhost driver. | ||
|
||
|
||
Use Cases | ||
--------- | ||
Guest instance users may benefit from increased network performance and | ||
throughput. | ||
|
||
|
||
Project Priority | ||
----------------- | ||
None | ||
|
||
|
||
Proposed change | ||
=============== | ||
|
||
In order to address the problem, an administrator should have the control | ||
over the paralleled packet processing by disabling the multiqueue support | ||
for certain workloads, where this feature may lead to a | ||
performance degradation. | ||
|
||
Introducing a new parameter to the image properties, for the users | ||
to control the virtio-net multiqueue feature, and be able to disable it, | ||
if desired. | ||
|
||
hw_vif_multiqueue_enabled=true|false (default false) | ||
|
||
Currently, the number of queues will match the number of vCPUs, defined for the | ||
instance. | ||
|
||
NOTE: Virtio-net multiqueue should be enabled in the guest OS manually, using | ||
ethtool. For example: | ||
|
||
ethtool -L <NIC> combined #num_of_queues | ||
|
||
Alternatives | ||
------------ | ||
|
||
Considering the limited amount of MSI vectors in the guest, | ||
users could have the ability to create instances with vNICs operating at | ||
various traffic rates, by controlling the use of the multiqueue feature. | ||
|
||
For example, one NIC might be used solely for administrative traffic | ||
and so will not need multiqueue enabled, while other NICs might be used for | ||
high throughput traffic and will require multiple queues. | ||
|
||
E.g. nova boot -nic net_id=<id>,queues=# | ||
|
||
However, currently, there is no agreement on the interface the users use to | ||
specify the number of queues per each vNIC. | ||
|
||
Another option would be to replace "queues=#" with "throughput=high|medium|low" | ||
which would set the number of queues to 100%|50%|1 of vCPUs. | ||
|
||
There will be a need for a following spec to determine the correct interface. | ||
|
||
Finally, the administrator could set the overall number of allowed queues on | ||
the flavor extra_specs. However, since the amount of queues affects only the | ||
guest resources, it should be up to the user to control the amount per each | ||
nic. | ||
|
||
|
||
Data model impact | ||
----------------- | ||
|
||
None | ||
|
||
REST API impact | ||
--------------- | ||
|
||
None. | ||
|
||
Security impact | ||
--------------- | ||
|
||
None. | ||
|
||
Notifications impact | ||
-------------------- | ||
|
||
None. | ||
|
||
Other end user impact | ||
--------------------- | ||
|
||
None. | ||
|
||
Performance Impact | ||
------------------ | ||
|
||
None | ||
|
||
Other deployer impact | ||
--------------------- | ||
|
||
None | ||
|
||
Developer impact | ||
---------------- | ||
|
||
None. | ||
|
||
Implementation | ||
============== | ||
|
||
Assignee(s) | ||
----------- | ||
Primary assignee: | ||
vromanso@redhat.com | ||
|
||
Work Items | ||
---------- | ||
|
||
* Update the vif config to set the correct number of vCPUs to | ||
virtio-net devices | ||
* Update the NetworkRequest object to add number of queues per port | ||
|
||
Dependencies | ||
============ | ||
|
||
None | ||
|
||
|
||
Testing | ||
======= | ||
|
||
Unit tests | ||
Requires Libvirt >= 1.0.5 | ||
|
||
Documentation Impact | ||
==================== | ||
|
||
The new parameters in flavor extra specs and the new network port property | ||
"queues" in nova boot command. | ||
Some recommendations for an effective usage, should be mentioned as well. | ||
|
||
|
||
References | ||
========== | ||
|
||
http://www.linux-kvm.org/page/Multiqueue | ||
|