Alternate forwarding planes: VPP

NetEasen edited this page Jun 6, 2018 · 9 revisions

Table of Contents

1. Introduction

This tutorial is a step-by-step guide showing how to build a high-performance software router using two complementary open source projects: VPP and FRRouting.

VPP is a feature rich, high-performance packet processing platform that runs on commodity hardware. VPP leverages DPDK[1] and runs as a Linux user-space application, moving the routing/forwarding of packets out of the Linux kernel.

FRRouting (FRR) is a multi-platform routing stack that provides implementations for BGP, OSPF, IS-IS, EIGRP, RIP, LDP, PIM and NHRP. FRR is a fork of Quagga.

More information about VPP and FRRouting can be found in [2] and [3].

This tutorial explains how to integrate the two projects together. In other words, how to build a router using FRR for the control plane and VPP for the data-plane. For this to work, we're going to use VPP's router plugin [4], which acts as a glue between VPP and any Linux-based routing stack. This plugin implements logic to punt control packets to the Linux network stack so routing daemons can work. Also, it implements a netlink-based mechanism that continually synchronizes the Linux's kernel routing table into VPP's FIB (the fast-path).

2. Setup

This section explains how to install VPP (with all the required plugins) and FRR.

For simplicity sake, we're going to build and install everything in a Vagrant VM provided in the VPP repository.

Before proceeding, make sure that the host operating system has the following software installed:

2.1. VPP

1 - Clone the VPP repository(the latest vpp version 18.04 can work fine with vppsb) and create the Vagrant guest machine (this can take some time):

$ git clone https://gerrit.fd.io/r/vpp
$ cd vpp/build-root/vagrant/
$ export VPP_VAGRANT_DISTRO=ubuntu1604
$ vagrant up

2 - SSH into the Vagrant VM:

$ vagrant ssh

From now on all commands should be typed in the VM.

3 - Clone the VPPsb repository:

$ cd ~
$ git clone https://gerrit.fd.io/r/vppsb

4 - Build and install VPP with the netlink and router plugins:

$ sudo apt-get install -y python-cffi python-pycparser
$ cd /vpp
$ ln -sf /home/vagrant/vppsb/netlink
$ ln -sf /home/vagrant/vppsb/router
$ ln -sf ../../netlink/netlink.mk build-data/packages/
$ ln -sf ../../router/router.mk build-data/packages/
$ cd build-root/
$ ./bootstrap.sh
$ make V=0 PLATFORM=vpp TAG=vpp_debug install-deb netlink-install router-install
$ sudo dpkg -i *.deb
$ sudo ln -sf \
        /vpp/build-root/install-vpp_debug-native/router/lib64/router.so.0.0.0 \
        /usr/lib/vpp_plugins/router.so

5 - Edit the default configuration file (/etc/vpp/startup.conf) as follows:

unix {
  nodaemon
  log /tmp/vpp.log
  cli-listen /run/vpp/cli.sock
  full-coredump
}

api-trace {
  on
}

api-segment {
  gid vpp
}

To test if the installation went ok, restart VPP with the following command:

$ sudo service vpp restart

In another shell, type the following:

$ sudo vppctl enable tap-inject

If both commands are executed without any errors, it means that VPP and the router plugin were installed successfully.

2.2. FRRouting

The following commands will install all FRR dependencies and then FRR itself using a standalone deb package:

$ sudo apt-get install iproute2 libc-ares2
$ cd ~
$ wget https://github.com/FRRouting/frr/releases/download/frr-3.0.3/frr_3.0.3-1_ubuntu16.04.1_amd64.deb
$ sudo dpkg -i frr*.deb

3. Testing

In this test we're going to use the GNS3 network simulator to set up the following topology:

GNS3 topology

In this topology, R1 and R3 are standard Cisco IOS-XE virtual routers. Any other virtual router can be used as long as it supports OSPFv2 (e.g. Juniper vMX). R2 is our VPP VM with FRR installed.

The goal is to set up OSPF in the network and then use the iperf traffic generator in the hosts to confirm that VPP (the fast-path) is being used in R2 to route the data packets.

Configuring GNS3

Open GNS3 and add a VirtualBox template for the VPP VM we created earlier. In the Network tab, specify that this VM should have 5 network adapters by default as shown in the image below:

In the GNS3 topology, use VPP's g3 interface to connect to R1 and VPP's g4 interface to connect to R2.

Configuring VPP

The first thing to do is to edit the dpdk section of the /etc/vpp/startup.conf file and specify the two interfaces we want to use with VPP. Example:

dpdk {
  dev 0000:00:0a.0
  dev 0000:00:10.0
}

We need to use the PCI ID of the NICs and not their Linux interface names. To map interface names to PCI IDs, we can use the following command:

# sudo lshw -class network -businfo
Bus info          Device     Class      Description
===================================================
pci@0000:00:03.0  enp0s3     network    82540EM Gigabit Ethernet Controller
pci@0000:00:08.0  enp0s8     network    82540EM Gigabit Ethernet Controller
pci@0000:00:09.0  enp0s9     network    82540EM Gigabit Ethernet Controller
pci@0000:00:0a.0  enp0s10    network    82540EM Gigabit Ethernet Controller
pci@0000:00:10.0  enp0s16    network    82540EM Gigabit Ethernet Controller

For this particular test, put the last two devices in the VPP configuration file. Also, make sure that the two associated Linux interfaces are administratively down:

# ifconfig enp0s10 down
# ifconfig enp0s16 down

Now restart VPP for the changes to take effect:

# service vpp restart

If everything went right, the interfaces we specified in the configuration file should disappear from the Linux kernel network stack. DPDK owns them now.

Note: VPP is generally used in conjunction with DPDK in order to have direct access to the NICs, bypassing the Linux network stack. VPP also supports two non-DPDK drivers, namely veth and tun, to enable use inside network namespaces. More information about using VPP with network namespaces can be found in [7].

Now configure the VPP data-plane interfaces:

# vppctl create loopback interface
# vppctl set interface state loop0 up
# vppctl set interface state GigabitEthernet0/a/0 up
# vppctl set interface state GigabitEthernet0/10/0 up
# vppctl set interface ip address loop0 2.2.2.2/32
# vppctl set interface ip address GigabitEthernet0/a/0 10.0.10.2/24
# vppctl set interface ip address GigabitEthernet0/10/0 10.0.20.2/24

Enable the router plugin with the following command:

# vppctl enable tap-inject

This command will create a Linux tap interface for each VPP data-plane interface. These tap interfaces are basically a bidirectional communication channel between VPP and the Linux network stack. They must be given the same IP addresses configured on their respective interfaces in VPP.

To do that, use the vppctl show tap-inject command to see the Linux/VPP interfaces mappings and then use iproute2 to configure the tap interfaces appropriately.

Example:

# vppctl show tap-inject
GigabitEthernet0/10/0 -> vpp1
GigabitEthernet0/a/0 -> vpp0
loop0 -> vpp2

# ip addr add 10.0.10.2/24 dev vpp0
# ip addr add 10.0.20.2/24 dev vpp1
# ip addr add 2.2.2.2/32 dev vpp2
# ip link set dev vpp0 up
# ip link set dev vpp1 up
# ip link set dev vpp2 up

Configuring FRR

Edit /etc/frr/zebra.conf as follows:

hostname zebra
password zebra
log stdout

Edit /etc/frr/ospfd.conf as follows:

hostname ospfd
password zebra
log stdout
!
router ospf
 network 2.2.2.2/32 area 0
 network 10.0.10.2/24 area 0
 network 10.0.20.2/24 area 0
!

Start zebra and ospfd:

# zebra -d
# ospfd -d

Configuring the IOS-XE routers

R1 configuration:

interface Loopback1
 ip address 1.1.1.1 255.255.255.255
 ip ospf 1 area 0
!
interface GigabitEthernet1
 ip address 10.0.10.1 255.255.255.0
 ip ospf 1 area 0
!
interface GigabitEthernet2
 ip address 192.168.10.1 255.255.255.0
 ip ospf 1 area 0
!
router ospf 1

R3 configuration:

interface Loopback1
 ip address 3.3.3.3 255.255.255.255
 ip ospf 1 area 0
!
interface GigabitEthernet1
 ip address 10.0.20.3 255.255.255.0
 ip ospf 1 area 0
!
interface GigabitEthernet2
 ip address 192.168.20.3 255.255.255.0
 ip ospf 1 area 0
!
router ospf 1

Verification

In the VPP VM, open vtysh and check if OSPF converged in the network:

# show ip ospf neighbor 

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
1.1.1.1           1 Full/DR           36.212s 10.0.10.1       vpp0:10.0.10.2           0     0     0
3.3.3.3           1 Full/Backup       34.631s 10.0.20.3       vpp1:10.0.20.2           0     0     0

The two OSPF adjacencies indicate that VPP is punting the OSPF packets to the tap interfaces as expected (this can be confirmed by looking at the Interface column).

Using the ip route command we can see that zebra installed the OSPF routes in the Linux kernel:

# ip route
default via 10.0.2.2 dev enp0s3 
1.1.1.1 via 10.0.10.1 dev vpp0  proto 188  metric 20 
3.3.3.3 via 10.0.20.3 dev vpp1  proto 188  metric 20 
10.0.2.0/24 dev enp0s3  proto kernel  scope link  src 10.0.2.15 
10.0.10.0/24 dev vpp0  proto kernel  scope link  src 10.0.10.2 
10.0.20.0/24 dev vpp1  proto kernel  scope link  src 10.0.20.2 
172.28.128.0/24 dev enp0s8  proto kernel  scope link  src 172.28.128.7 
172.28.128.0/24 dev enp0s9  proto kernel  scope link  src 172.28.128.8 
192.168.10.0/24 via 10.0.10.1 dev vpp0  proto 188  metric 20 
192.168.20.0/24 via 10.0.20.3 dev vpp1  proto 188  metric 20

As it was mentioned before, the VPP router plugin also takes care of synchronizing the Linux routing table into VPP's FIB. We can confirm this using the vppctl show ip fib command:

# vppctl show ip fib
Table 0, fib_index 0, flow hash: src dst sport dport proto 
     Destination         Packets          Bytes         Adjacency   
192.168.10.0/24                    0               0 weight 1, index 16, multipath
                                                     GigabitEthernet0/a/0
                                                     IP4: 08:00:27:d5:0e:67 -> ca:03:47:f1:00:00 shared 3
192.168.20.0/24                    0               0 weight 1, index 18, multipath
                                                     GigabitEthernet0/10/0
                                                     IP4: 08:00:27:bd:6c:4b -> ca:02:3f:4d:00:1d shared 3
224.0.0.0/24                     113            8632 weight 1, index 8
                                                      unknown 14
1.1.1.1/32                         0               0 weight 1, index 16, multipath
                                                     GigabitEthernet0/a/0
                                                     IP4: 08:00:27:d5:0e:67 -> ca:03:47:f1:00:00 shared 3
2.2.2.2/32                         0               0 weight 1, index 13
                                                      2.2.2.2/32
3.3.3.3/32                         0               0 weight 1, index 18, multipath
                                                     GigabitEthernet0/10/0
                                                     IP4: 08:00:27:bd:6c:4b -> ca:02:3f:4d:00:1d shared 3
10.0.10.0/24                       0               0 weight 2, index 10, multipath
                                                      10.0.10.2/24
10.0.10.1/32                       0               0 weight 1, index 14
                                                     GigabitEthernet0/a/0
                                                     IP4: 08:00:27:d5:0e:67 -> ca:03:47:f1:00:00
10.0.10.2/32                      26            2716 weight 1, index 7
                                                      10.0.10.2/24
10.0.20.0/24                       0               0 weight 1, index 11
                                                      10.0.20.2/24
10.0.20.2/32                      17            1630 weight 1, index 12
                                                      10.0.20.2/24
10.0.20.3/32                       0               0 weight 1, index 17
                                                     GigabitEthernet0/10/0
                                                     IP4: 08:00:27:bd:6c:4b -> ca:02:3f:4d:00:1d

Now let's generate some data traffic between the two hosts using the iperf tool.

On Host 1 (server):

# iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.100 port 5001 connected with with 192.168.20.100 port 51412
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec  1.19 MBytes  1.00 Mbits/sec   0.007 ms    0/  852 (0%)

On Host 2 (client):

# iperf -u -c 192.168.10.100 -b 1M
------------------------------------------------------------
Client connecting to 192.168.10.100, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.20.100 port 51412 connected with 192.168.10.100 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.19 MBytes  1000 Kbits/sec
[  3] Sent 852 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  1.19 MBytes  1.00 Mbits/sec   0.007 ms    0/  852 (0%)

There it is: VPP is routing the data packets using OSPF learned routes. At this point we can say that FRR and VPP were successfully integrated.

But how can we be sure that it's VPP that is routing the packets and not the Linux kernel?

There are several possible ways to check that. For example:

  • Disabling IP routing in the Linux kernel with the sysctl -w net.ipv4.ip_forward=0 command;
  • Checking the Rx/Tx counters from Linux/VPP using the ifconfig/vppctl show interfaces commands;
  • Using tcpdump in the VPP tap interfaces to verify that only OSPF packets are being punted to the Linux network stack;
  • Using VPP's Trace Tools as shown in [6] and [7].

In this topology we used OSPF but we could have used other routing protocols like BGP and RIP. Minor modifications in the VPP router plugin would be necessary in order to support IS-IS and EIGRP.

4. References

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.