Alexander Ostapenko edited this page Nov 1, 2018 · 12 revisions

Clouds use virtualization technologies in their operation. Tempesta FW can work in a virtualized environment, so it can be used in clouds. Since Tempesta FW extends Linux operating system, it's sensitive to type of the virtualization environment. The doc describes Tempesta FW use cases in different virtual environments.

Hardware virtualization

Examples of hardware virtualization are KVM and Xen in HVM mode. Amazon EC2 uses this type of virtualization for HVM AMIs. A guest operating system isn't aware about vertualized environment, so the case is the most straightforward to deploy and use. In fact QEMU is the most used virtualization by our development team. We also have tested Tempesta FW usage at Amazon EC2 M4 instance. Since guest OS kernel is totally isolated from the host, hardware virtualization provides the most reliability and security at the cost of performance however.

Private cloud

Tempesta FW can be deployed in either a private cloud or a public cloud using hardware virtualization. In private cloud you can deploy Tempesta FW at the host system, so that it will accelerate and protect Web applications running in VMs.

Public cloud

If a public cloud uses KVM, then you can deploy Tempesta FW in a VM to serve Web services in other VMs. Note that you still can use common HTTP server (e.g. Apache HTTPD) in the same VM with Tempesta FW. You can use the same deployment scenario in private cloud as well.

Note: Tempesta FW requires SSE 4.2 and AVX2, so ensure that the public cloud instance is running on top of appropriate hardware. Amazon EC2 uses different hardware depending on an instance type, the instance types and their hardware can be found here.

Performance

While using Tempesta FW inside VM (with hardware virtualization via KVM) it is worth consider some issues regarding performance accounting. In particular, these issues are noticeable during interrupts processing (including interprocess interrupts - IPI), writing to special registers (MSR) and other events occurred during guest code execution, which cause transitions between guest and hypervisor modes. To better understand these issues, some basic details (regarding hardware virtualization as whole) must be covered. Hardware support for virtualization in Intel processors is provided by a operations called VMX operations. There are VMX root operation (which hypervisor run in) and VMX non-root operation (for guest software). VM-entry transitions is VMX root => VMX non-root. VM-exit transitions is VMX non-root => VMX root. In VMX non-root mode certain instructions and events cause VM-exits to the hypervisor. VMX non-root operation and VMX transitions are controlled by a data structure called a virtual-machine control structure (VMCS). A hypervisor could use a different VMCS for each virtual machine (VM) that it supports. For a VMs with multiple processors (VCPUs), QEMU-KVM hypervisor use different VMCS for each VCPU.

If, for example, consider IPI processing in VM guest - on Intel processors with x2APIC support, IPI generation is simply write operation into some address of model specific registers (MSR), which (in QEMU-KVM hypervisor) causes a VM-exit on IPI source CPU. IPI receiving also causes VM-exit - at least on Intel processors without posted-interrupt (so called APICv) support.

At the same time, KVM uses domain model for performance monitoring counters (PMC) virtualization, which implies saving and restoring of relevant PMC registers only while execution switching between different guests, and not save/restore PMC registers during VM-exit/VM-entry in context of one current guest. So, in case of IPI receiving on target CPU we collect PMC not only for guest's operations, but also for hypervisor operations (between VM-exit and VM-entry). As a result, performance measurement for IPI-handler inside guest (e.g. via perf) will account not only IPI-handler processing in guest but also IPI-handler processing on host (on behalf of guest IPI) on target CPU. This effect, may appear not only during IPI processing, but also in other guest operations, which require VM-exit into hypervisor.

Processing of interrupts in non-root mode (on Intel processors with VMX supported) can be accelerated in KVM, if processor has virtualized APIC support (APICv, including posted-interrupt). In this case on target CPU, which receives interrupt in non-root mode, VM-exit does not occur and processor will pass interrupt to the guest on its own, without hypervisor intervention. APICv support can be checked in special Capability Reporting VMX registers of MSR; the following bits must be set for Intel 64 architecture:

  • Activate secondary controls (bit 63 of IA32_VMX_PROCBASED_CTLS)
  • Virtualize APIC accesses (bit 32 of IA32_VMX_PROCBASED_CTLS2)
  • APIC-register virtualization (bit 40 of IA32_VMX_PROCBASED_CTLS2)
  • Virtual-interrupt delivery (bit 41 of IA32_VMX_PROCBASED_CTLS2)
  • Process posted interrupts (bit 39 of IA32_VMX_TRUE_PINBASED_CTLS if bit 55 is set in IA32_VMX_BASIC; otherwise - bit 39 of IA32_VMX_PINBASED_CTLS)

besides, kernel must be built with CONFIG_X86_LOCAL_APIC enabled and kvm_intel kernel module must be loaded with parameter enable_apicv = Y (default: N). The values of specified bits can be read via rdmsr utility (package msr-tools):

rdmsr [options] <register_address>

Registers specified above have following addresses:

  • IA32_VMX_BASIC: 0x480
  • IA32_VMX_PROCBASED_CTLS: 0x482
  • IA32_VMX_PROCBASED_CTLS2: 0x48B
  • IA32_VMX_PINBASED_CTLS: 0x481
  • IA32_VMX_TRUE_PINBASED_CTLS: 0x48D

For automation of cumbersome process of bits calculation - perl script can be used.

Detailed information about Capability Reporting VMX registers and their connection with VM-execution control can be found in Intel® 64 and IA-32 Architectures Software Developer’s Manual: Volume 3 (Sections 35.1, 24.6.1, 24.6.2, Appendix A.3).

Regarding of Tempesta FW performance inside VM - also VCPU-CPU mapping may be considered. In QEMU-KVM hypervisor each VM is separate QEMU process on host. Each VCPU (in VMX/KVM) - is a thread of corresponding QEMU process. These threads are distributed on host CPUs and between them by host Linux scheduler like any other usual thread/process in OS. In default configuration, VCPUs of virtual machine are not pinned to any CPU and can be distributed between all of them (regardless of VCPUs count specified in -smp QEMU option). E.g. if we defined two VCPU for guest VM on the host with four CPUs, then any of these two VCPU can be executed on any of host CPUs. Besides, due to free distribution between all host CPUs, situations with several VCPUs on one CPU processing often could occur.

To avoid this default behavior there is variant of pinning guest's all VCPUs to host's CPUs (e.g. via libvirt command virsh vcpupin <VM> <VCPU> <CPU>). However, result of such pinning, during Tempesta FW testing under load, demonstrates the same performance (at least in 'requests per second'value) compared against not pinned VCPUs. It seems that host Linux scheduler's balancer effectively distribute not pinned VCPUs (QEMU threads) between host CPUs; pinning is necessary only when some workloads are already pinned for particular host CPUs, so we need manually distribute (pin) all workloads in this case - to gain maximum performance.

Migration

A KVM virtual machine with running Tempesta FW can be migrated to other host system. Live migration can be used to minimize the downtime. See KVM migration page for more information.

Paravirtualization

Probably the most famous example of paravirtualization is Xen. Whie the most Amazon EC2 instances are HVMs, there are still several paravirtualization instances, PV AMIs.

Technically you can use Tempesta FW in paravirtualized environment, either at domU or dom0 layer in terms of Xen, just the same way as for hardware virtualization case. However, Paravirtualization requires OS kernel awareness about running in paravirtualized environment, so you need to build your Linux kernel with configured paravirtualization.

Amazon EC2 PV AMI

To run Tempesta FW in Amazon EC2 PV instance, you need to build Tempesta FW's kernel according to Xen's kernel config. Please use following guide to load the Xen guest kernel with Tempesta FW support in Amazon EC2 PV AMI.

Containers

The popular examples of container virtualization are LXC (Linux Containers), OpenVZ and its commercial variant Virtuozzo.

All guest containers and the host system share the same Linux kernel, so Tempesta FW must be installed on the host system. In this case Tempesta FW can proxy ingress traffic to hosted containers, just like the private cloud case.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.