System Architecture
Computer networks have traditionally been built on the end-to-end principle or the principle of transparency [Saltzer]. It implies that all activities that are not connected directly with packet forwarding, should take place on endpoints - on the same servers where the application that consumes the packets is running. It is assumed that all the packets sent to the network will be received in the same form from the network and thus the network will be completely transparent to the user.
However, in order to meet the principle of transparency, the endpoint devices must be sufficiently advanced: perform forwarding of missing packets themselves, filter traffic, meter traffic, etc. With the development of the network and increase of the complexity of the topology, due to duplication, administration tasks and other issues, e.g. load balancing, it becomes more convenient to take out such functionality from the terminal devices and move them inside the network, calling them network functions.
Originally, network functions were performed on standard servers. However, network functions typically have more specialized performance requirements than terminal devices, because the amount of traffic that goes through it is much higher. Commodity server means architecture suitable for a wide range of tasks and utilizing the general purpose operating system, which in turn imposes performance limitations on the network function.
E.g. let's look at the simplified steps for receiving a packet on the example of Linux OS.
Table 1 shows steps for processing a single packet:
- four moves using DMA - Direct Memory Access (steps 2, 6, 21, 22),
- two copies
- one cache write
- four context switches
- two interrupt processing
The Linux operating system assumes a complete parse of the packet. This means filling in about forty fields in the sk_buf structure responsible for packet metadata. Although these structures are allocated in memory in advance, a large amount of
| Step | Description | Stage |
|---|---|---|
| 1 | NIC takes a packet from a network wire | |
| 2 | NIC writes a packet into the NIC ring reception buffer | NIC |
| 3 | NIC driver initiates "new packet" interrupt | |
| 4 | OS reacts to an interrupt by starting package processing | |
| 5 | OS initializes sk_buf structure with information about the package | |
| 6 | OS moves the packet to socket buffer in DMA memory | DMA |
| 7 | Primary processing: checksum calculation, decryption | |
| 8 | Cache miss - packet data not in the cache. The packet is placed in the cache | Cache |
| 9 | OS processes the packet with network stack, fills protocols | |
| 10 | User calls system call "read" | |
| 11 | Switching the context from user to OS | |
| 12 | OS copies the package to the COPY user buffer | COPY |
| 13 | Switch context from OS to user | |
| 14 | The user can process the package | |
| 15 | The user calls the system call "write" | |
| 16 | Switching the context from user to OS | |
| 17 | OS copies the packet to socket buffer in COPY memory | COPY |
| 18 | The NIC driver writes the packet to the DMA ring transmission buffer | DMA |
| 19 | NIC transmits the packet to the NIC network wire | NIC |
| 20 | The NIC driver initiates an interruption of the "package sent" | |
| 21 | OS reacts to interrupt, switch to user |
Table 1 - Steps to receive-send a packet
time is spent on initializing them before processing each package.
The processing itself is often redundant for an application that already knows what type of packages it receives. Thus, items 5 and 9 are expensive and redundant.
Let's consider how much time the network function has to process a packet. The size of an Ethernet packet according to the standard can vary from 64 bytes to 1518 bytes. Let's take the average size of packets as S = 350 bytes. Let's consider receiving/transmitting such packets at the speed V = 40Gb/s. To process one packet we spend:
- 𝑁 = 𝑇/𝑉 * 𝑆 = 10^9ns/(40 * 1024^3/8)bytes * 350 bytes = 65ns
Or about 162 processor cycles per packet on a processor with frequency µ = 2.5 GHz. During this time, it is necessary to perform all of the above operations and apart from these overheads, process the packet itself.
As a result of these problems, network functions began to be executed on specialized equipment called a middlebox [Middlebox]. In this case, the hardware/software system is designed to effectively perform single specific function, which allows to optimize it for efficiency. Middleboxes has solved the performance problem in general and are widely used now as the standard for building a telecommunication network.
However, due to the problems associated with them (cost, inflexibility, closeness, etc.) different ways to execute network functions on the commodity hardware are being developed, e.g. see [Dixon].
But the first step to achieve this is to solve the problem of low performance when interacting with the network.