Virtual Routing and Forwarding (VRF)

The VRF documentation in the Linux kernel source already provides a comprehensive overview of VRFs and extensive iproute2 usage examples. Therefore, this section does not repeat everything written there. Instead, it will focus on the issues specific to VRF offload using a basic example of two offloaded VRFs on top of the switch ports.

Table of Contents

1. VRF Configuration
   1. FIB Rules
   2. Inter-VRF Routing
2. Further Resources

VRF Configuration

The following example shows two VRF devices configured on top of the switch ports:

            +------------------------+                   +------------------------+
            |  vrf-blue (table 10)   |                   |   vrf-red (table 20)   |
            +------------------------+                   +------------------------+
               |                  |                         |                  |
           +-------+          +-------+                 +-------+          +-------+
           | sw1p3 |          | sw1p5 |                 | sw1p4 |          | sw1p6 |
           +-------+          +-------+                 +-------+          +-------+
         192.168.100.1/24   192.168.101.1/24          192.168.100.1/24   192.168.200.1/24

To create the VRF devices, run:

$ ip link add name vrf-blue type vrf table 10
$ ip link set dev vrf-blue up
$ ip link add name vrf-red type vrf table 20
$ ip link set dev vrf-red up

Set a default route in each table to prevent lookup from leaking into other tables:

$ ip [-6] route add table 10 unreachable default
$ ip [-6] route add table 20 unreachable default

Note: This is mainly important for packets forwarded by the kernel. The device, upon route miss will drop the packet.

Then, enslave the ports to these devices like you would any other master device (e.g. bridge, bond):

$ ip link set dev sw1p3 master vrf-blue
$ ip link set dev sw1p5 master vrf-blue
$ ip link set dev sw1p4 master vrf-red
$ ip link set dev sw1p6 master vrf-red

Addresses can be configured on the interfaces either before or after they are enslaved to the VRF device:

$ ip address add 192.168.100.1/24 dev sw1p3
$ ip address add 192.168.101.1/24 dev sw1p5
$ ip address add 192.168.100.1/24 dev sw1p4
$ ip address add 192.168.200.1/24 dev sw1p6

Upon enslavement to a VRF device, local and connected routes are automatically moved to the VRF's table from the local and main tables, respectively:

$ ip [-6] route show table 10 dev sw1p3
broadcast 192.168.100.0  proto kernel  scope link  src 192.168.100.1
192.168.100.0/24  proto kernel  scope link  src 192.168.100.1 offload
local 192.168.100.1  proto kernel  scope host  src 192.168.100.1
broadcast 192.168.100.255  proto kernel  scope link  src 192.168.100.1

FIB Rules

The kernel uses FIB rules (Policy Based Routing, PBR) to direct a packet's lookup through the different FIB tables. By default, packets first undergo a lookup in the local table and in case of a miss in the main table as well.

$ ip [-6] rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Upon the creation of the first VRF device, an FIB rule is inserted with priority 1000 to direct lookup of packets passing through enslaved devices to the VRF's table:

$ ip link add name vrf-blue type vrf table 10
$ ip [-6] rule show
0:      from all lookup local
1000:   from all lookup [l3mdev-table]
32766:  from all lookup main
32767:  from all lookup default

Therefore, when packets are forwarded by the kernel they first undergo a lookup in the local table and then in the VRF's table. This is in contrast to the device's data path, where packets only perform a lookup in the VRF's table.

It is thus recommended to change the order of the default rules, so that the rule for the local table is just before the main table:

$ ip [-6] rule add pref 32765 table local
$ ip [-6] rule del pref 0
$ ip [-6] rule show
1000:   from all lookup [l3mdev-table]
32765:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Note: Prior to kernel 4.8, iif and oif FIB rules were needed for each VRF device, instead of a single l3mdev rule for all VRF devices. The old model is not supported and upon the insertion of such rules the device's FIB tables are flushed and forwarding is performed by the kernel.

Inter-VRF Routing

There are two ways to perform inter-VRF routing. The first method involves a route configured in one VRF's table pointing to a device enslaved to a different VRF. Using the example from above:

$ ip route add table 10 192.168.200.0/24 dev sw1p6

Packets hitting these routes can be forwarded by the device:

$ ip route show table 10
...
192.168.200.0/24 dev sw1p6  scope link offload

The second way to perform inter-VRF routing is to point the route to a different VRF master device. Packets hitting such routes will undergo a full lookup in the VRF's table:

$ ip route add table 10 192.168.200.0/24 dev vrf-red

However, these routes are not currently offloaded. Packets hitting them are forwarded by the kernel instead:

$ ip route show table 10
...
192.168.200.0/24 dev vrf-red  scope link

Note that the offload flag is not set.

Further Resources

1. "VRF Tutorial" by David Ahern, Netdev 1.1 (slides, video)
2. "What is an L3 Master Device?" by David Ahern, Netdev 1.2