Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Clouds use virtualization technologies in their operation. Tempesta FW can work in a virtualized environment, so it can be used in clouds. Since Tempesta FW extends Linux operating system, it's sensitive to type of the virtualization environment. The doc describes Tempesta FW use cases in different virtual environments.
Examples of hardware virtualization are KVM and Xen in HVM mode. Amazon EC2 uses this type of virtualization for HVM AMIs. A guest operating system isn't aware about vertualized environment, so the case is the most straightforward to deploy and use. In fact QEMU is the most used virtualization by our development team. We also have tested Tempesta FW usage at Amazon EC2 M4 instance. Since guest OS kernel is totally isolated from the host, hardware virtualization provides the most reliability and security at the cost of performance however.
Tempesta FW can be deployed in either a private cloud or a public cloud using hardware virtualization. In private cloud you can deploy Tempesta FW at the host system, so that it will accelerate and protect Web applications running in VMs.
If a public cloud uses KVM, then you can deploy Tempesta FW in a VM to serve Web services in other VMs. Note that you still can use common HTTP server (e.g. Apache HTTPD) in the same VM with Tempesta FW. You can use the same deployment scenario in private cloud as well.
Note: Tempesta FW requires SSE 4.2 and AVX2, so ensure that the public cloud instance is running on top of appropriate hardware. Amazon EC2 uses different hardware depending on an instance type, the instance types and their hardware can be found here.
While using Tempesta FW inside VM (with hardware virtualization via KVM) it is worth
consider some issues regarding performance accounting. In particular, these issues are
noticeable during interrupts processing (including interprocess interrupts - IPI), writing
to special registers (MSR) and other events occurred during guest code execution, which
cause transitions between guest and hypervisor modes. To better understand these issues,
some basic details (regarding hardware virtualization as whole) must be covered. Hardware
support for virtualization in Intel processors is provided by a operations called
operations. There are
VMX root operation (which hypervisor run in) and
operation (for guest software). VM-entry transitions is
VMX root => VMX non-root.
VM-exit transitions is
VMX non-root => VMX root. In
VMX non-root mode certain
instructions and events cause VM-exits to the hypervisor. VMX non-root operation and
VMX transitions are controlled by a data structure called a virtual-machine control
VMCS). A hypervisor could use a different VMCS for each virtual machine (VM)
that it supports. For a VMs with multiple processors (VCPUs), QEMU-KVM hypervisor use
different VMCS for each VCPU.
If, for example, consider IPI processing in VM guest - on Intel processors with
support, IPI generation is simply write operation into some address of model specific
registers (MSR), which (in QEMU-KVM hypervisor) causes a VM-exit on IPI source CPU.
IPI receiving also causes VM-exit - at least on Intel processors without
At the same time, KVM uses domain model for
performance monitoring counters
(PMC) virtualization, which implies saving and restoring of relevant PMC registers
only while execution switching between different guests, and not save/restore PMC
registers during VM-exit/VM-entry in context of one current guest. So, in case of
IPI receiving on target CPU we collect PMC not only for guest's operations, but also
for hypervisor operations (between VM-exit and VM-entry). As a result, performance
measurement for IPI-handler inside guest (e.g. via
perf) will account not only
IPI-handler processing in guest but also IPI-handler processing on host (on behalf
of guest IPI) on target CPU. This effect, may appear not only during IPI processing,
but also in other guest operations, which require VM-exit into hypervisor.
Processing of interrupts in non-root mode (on Intel processors with VMX supported)
can be accelerated in KVM, if processor has virtualized APIC support (
posted-interrupt). In this case on target CPU, which receives interrupt
in non-root mode, VM-exit does not occur and processor will pass interrupt to the
guest on its own, without hypervisor intervention. APICv support can be checked in
Capability Reporting VMX registers of MSR; the following bits must be set
Intel 64 architecture:
- Activate secondary controls (bit 63 of
- Virtualize APIC accesses (bit 32 of
- APIC-register virtualization (bit 40 of
- Virtual-interrupt delivery (bit 41 of
- Process posted interrupts (bit 39 of
IA32_VMX_TRUE_PINBASED_CTLSif bit 55 is set in
IA32_VMX_BASIC; otherwise - bit 39 of
besides, kernel must be built with
CONFIG_X86_LOCAL_APIC enabled and
kernel module must be loaded with parameter
The values of specified bits can be read via
rdmsr utility (package
rdmsr [options] <register_address>
Registers specified above have following addresses:
For automation of cumbersome process of bits calculation - perl script can be used.
Detailed information about
Capability Reporting VMX registers and
their connection with VM-execution control can be found in
Intel® 64 and IA-32 Architectures Software Developer’s Manual: Volume 3 (Sections 35.1, 24.6.1, 24.6.2, Appendix A.3).
Regarding of Tempesta FW performance inside VM - also VCPU-CPU mapping may be
considered. In QEMU-KVM hypervisor each VM is separate QEMU process on host. Each
VCPU (in VMX/KVM) - is a thread of corresponding QEMU process. These threads are
distributed on host CPUs and between them by host Linux scheduler like any other
usual thread/process in OS. In default configuration, VCPUs of virtual machine are
not pinned to any CPU and can be distributed between all of them (regardless of VCPUs
count specified in
-smp QEMU option). E.g. if we defined two VCPU for guest VM on
the host with four CPUs, then any of these two VCPU can be executed on any of host
CPUs. Besides, due to free distribution between all host CPUs, situations with several
VCPUs on one CPU processing often could occur.
To avoid this default behavior there is variant of pinning guest's all VCPUs to host's
CPUs (e.g. via
virsh vcpupin <VM> <VCPU> <CPU>). However, result
of such pinning, during Tempesta FW testing under load, demonstrates the same
performance (at least in
'requests per second'value) compared against not pinned VCPUs.
It seems that host Linux scheduler's balancer effectively distribute not pinned VCPUs
(QEMU threads) between host CPUs; pinning is necessary only when some workloads are
already pinned for particular host CPUs, so we need manually distribute (pin) all
workloads in this case - to gain maximum performance.
A KVM virtual machine with running Tempesta FW can be migrated to other host system. Live migration can be used to minimize the downtime. See KVM migration page for more information.
Technically you can use Tempesta FW in paravirtualized environment, either at domU or dom0 layer in terms of Xen, just the same way as for hardware virtualization case. However, Paravirtualization requires OS kernel awareness about running in paravirtualized environment, so you need to build your Linux kernel with configured paravirtualization.
Amazon EC2 PV AMI
To run Tempesta FW in Amazon EC2 PV instance, you need to build Tempesta FW's kernel according to Xen's kernel config. Please use following guide to load the Xen guest kernel with Tempesta FW support in Amazon EC2 PV AMI.
All guest containers and the host system share the same Linux kernel, so Tempesta FW must be installed on the host system. In this case Tempesta FW can proxy ingress traffic to hosted containers, just like the private cloud case.