An IPv4 router over netmap for FreeBSD
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
INSTALL added space Oct 18, 2015
Makefile DPDK IP checksum experiments Oct 26, 2015
README minor edit Oct 6, 2015
TODO slight changes to TODO Nov 26, 2015
arp.c Initialize variable otherwise we can return an uninitialized value. Dec 2, 2015
arp.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
cleanup.c minor whitespace editing Oct 19, 2015
cleanup.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
cli.c style changes, minor cleanup of whitespace Oct 19, 2015
cli.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
config.c fix LP -> LLC on Rubicon Copyright Oct 18, 2015
config.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
counters.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
ether.c Remove a debug message that should not be there... Dec 6, 2015
ether.h Make ether_bridge() return an error when all the tx rings are full. Dec 5, 2015
event.c fix LP -> LLC on Rubicon Copyright Oct 18, 2015
event.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
icmp.c fix LP -> LLC on Rubicon Copyright Oct 18, 2015
icmp.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
if.c Add the basic support to connect network interfaces via VALE switches. Dec 6, 2015
if.h Add the basic support to connect network interfaces via VALE switches. Dec 6, 2015
inet.c Remove dead assignments. Dec 2, 2015
inet.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
ip.c Only count forwarded packets when there is no error. Dec 4, 2015
ip.h DPDK IP checksum experiments Oct 26, 2015
net.c fix LP -> LLC on Rubicon Copyright Oct 18, 2015
net.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
netmap-fwd.c Fix indentation. Dec 5, 2015
netmap-fwd.conf initial checkin Aug 28, 2015
netmap-fwd.pdf Add INSTALL, fix stats to be unsigned Oct 18, 2015
netmap.c Add the basic support to connect network interfaces via VALE switches. Dec 6, 2015
netmap.h fix LP -> LLC on Rubicon Copyright Oct 18, 2015
radix.c Commit the new version of netmap-fwd. Oct 6, 2015
radix.h initial checkin Aug 28, 2015
util.c Fix the log timer. Make sure it logs repeated messages once every 5 s… Dec 6, 2015
util.h Do not flood the console with log messages, count the equal messages … Dec 4, 2015

README

netmap-fwd was designed to require little to no setup.

Consider the following case:

[Test1]<---->[Router]<----->[Test2]<--->[Internet]

Where:

Test1:

dwc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=80008<VLAN_MTU,LINKSTATE>
       ether 02:13:06:c3:52:48
       inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

With a default route to 10.0.0.1

Test2:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
       ether e8:40:f2:c2:98:00
       inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

There is a static route to the 10.0.0.0/24 network:
  route to: 10.0.0.0
destination: 10.0.0.0
      mask: 255.255.255.0
   gateway: 192.168.0.88
       fib: 0
 interface: em0
     flags: <UP,GATEWAY,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
      0         0         0         0      1500         1         0

Router:

igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
       ether 00:08:a2:09:5b:4b
       inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
       ether 00:08:a2:09:5b:4d
       inet 192.168.0.88 netmask 0xffffff00 broadcast 192.168.0.255
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

With a default route to 192.168.0.10.

With this setup Test1 can reach Test2 and vice-versa. Test1 can also reach the internet via Test2.

With a working setup, like this one, if I want to replace the in-kernel routing of Router with netmap-fwd, all you have to do is:

# ./netmap-fwd igb3 igb5 &
[1] 5906
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       ether: 00:08:a2:09:5b:4b
       inet 10.0.0.1 netmask 255.255.255.0 broadcast 10.0.0.255
switching interface igb3 to netmap mode.
igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       ether: 00:08:a2:09:5b:4d
       inet 192.168.0.88 netmask 255.255.255.0 broadcast 192.168.0.255
switching interface igb5 to netmap mode.

netmap-fwd reads the interface IPs (it only works with IPv4 at this
point) and setup everything it needs.  There is a simple CLI to check
ARP and routes:

route show
Routing tables

Internet:
Destination        Gateway            Flags        Netif
0.0.0.0/0          192.168.0.10       UGS          igb5
10.0.0.0/24                           U            igb3
10.0.0.1                              UH           igb3
192.168.0.0/24                        U            igb5
192.168.0.88                          UH           igb5
arp
? (10.0.0.2) at 02:13:06:c3:52:48 on igb3 expires 1182 seconds
? (192.168.0.10) at e8:40:f2:c2:98:00 on igb5 expires 1181 seconds
? (192.168.0.88) at 00:08:a2:09:5b:4d on igb5 permanent
? (10.0.0.1) at 00:08:a2:09:5b:4b on igb3 permanent

And that's all. You're ready to go.

It was designed this way so
	1) It is easy to compare netmap-fwd with in-kernel routing, 
	2) if you kill netmap-fwd the in-kernel routing will continue to work without any change in your configuration.

But as you can see, the are a lot of room for improvements (IPv6, bgpd daemon fib synchronization and a few other details).

netmap-fwd accepts any number of network interfaces which may also include vlans.

Using pkt-gen, on a C2518 (quad core 1.7GHz Atom) 
654.302285 main_thread [1620] 1.175 Mpps (1.176 Mpkts 602.256 Mbps in 1001150 usec) 3.56 avg_batch
655.302630 main_thread [1620] 1.175 Mpps (1.175 Mpkts 601.769 Mbps in 1000345 usec) 3.55 avg_batch
656.304142 main_thread [1620] 1.175 Mpps (1.177 Mpkts 602.478 Mbps in 1001512 usec) 3.55 avg_batch
657.304823 main_thread [1620] 999.767 Kpps (1.000 Mpkts 512.229 Mbps in 1000681 usec) 3.53 avg_batch
[…]
Received 25618109 packets 1639558976 bytes 7225406 events 64 bytes each in 21.80 seconds.
Speed: 1.175 Mpps Bandwidth: 601.568 Mbps (raw 827.156 Mbps). Average batch: 3.55 pkts

FreeBSD can't do more than 233Kpps on the same hardware (single IP flow)

A bit more data:

Device under test				Network interface	Kernel forwarding	Fastforward enabled	netmap-fwd 
C2358 (2 core, 1.7 GHz, 4 GB RAM)		Intel I354		123 kpps		217 kpps		945 kpps
C2758 (8 core, 2.4 GHz, 8 GB RAM)		Chelsio T520 10G	270 kpps		426 kpps		1.683 Mpps
Xeon-D 1540 (8 core, 2 GHz, 32 GB RAM)		Intel X552 10G		439 kpps		557 kpps		2.230 Mpps
Xeon E3-1275 (4 core, 3.5 GHz, 32 GB RAM)	Intel X520-2 10G	1.058 Mpps		1.331 Mpps		5.053 Mpp

Luiz