RFC: Add testing for capture (pf_ring/afpacket/pcap/pcapgo/...)

[notti]

I wondered if it would be possible to make network capture/sending out packets testable on travis-ci. So I tried out some stuff and came up with this proof of concept. There is also Travis-ci output.

What is needed?

• precompiled linux kernel
• for pf_ring precompiled pf_ring kernel module

How does it work?

On travis-ci we can not load kernel modules (for security reasons) so the solution for this is to use qemu and run our own linux. Since kvm is not available this is a bit slower - but not that bad (booting + a short test took just a second).

To make everything fast and compact this setup does the following things:

• The kernel is stripped down to just the stuff that is needed (virtio-net, virtio-serial, serial-port, networking, initrd) and everything else is deactivated (e.g. all the filesystems except devtmpfs, all other device drivers, ...)
• kernel precompiled
• No init-system or anything - just the tester guest go application

Steps that get executed:

1. Main go-program (tester.go) builds the guest program (init/test.go)
2. It builds an initrd containing
   • pf_ring.ko
   • init
   • interp of init (ld - determined from the elf header of init)
   • needed libraries (determined from ld)
3. Create a socket for a control channel
   • Upon receiving a message from the control channel an answer is sent back
4. Create a socket for a network device
   • Upon receiving a packet from the network device an answer is sent back
5. Run qemu

Inside the guest init does the following:

1. mount devtmpfs
2. write test to the control channel
3. read a message from the control channel
4. load pf_ring kernel module
5. bring the network interface up and assign an ip address
6. write a packet to the network
7. read a packet from the network

The network device on the host works via a socket provided to qemu. The packet format on this socket is a big endian uint32 with the packet size followed by the network packet. Packets sent to this socket appear in the network device in the guest - and packets sent to the network device appear on the host in the socket.

The control socket can be used for synchronization (on the host it is a socket, and on the guest a char-device).

Alternatives

Travis-ci can be tricked into loading veth by requesting docker in travis.yml. With this We can create our private network devices and use those for testing. But with this only methods compiled in the travis-ci image can be tested (pcapgo, pcap should also work, af_packet I'm not sure, pf_ring definitely not).

Open questions

• Is this a viable solution?
• Where should the kernel image/modules be hosted (at the moment ~2MB)?
• Where should the testing code/kernel building script/config be hosted?

[gconnell]

For another project, I wrote https://github.com/google/stenographer/blob/master/integration_test/test.sh. It creates dummy interfaces and sends/receives traffic on them. It works fine, just requires sudo:required (https://github.com/google/stenographer/blob/master/.travis.yml#L4)

In short, it:

• pulls down a big .pcap file and tcpreplay
• builds the system-under-test
• creates a dummy interface with:
  • DUMMY="${DUMMY-dummy0}"
  • sudo /sbin/modprobe dummy
  • sudo ip link add $DUMMY type dummy || Error "$DUMMY may already exist"
  • sudo ifconfig $DUMMY promisc up
• starts system-under-test
• replays packets with
  • sudo tcpreplay -i $DUMMY --topspeed $BASEDIR/steno_integration_test.pcap

[gconnell]

The above should work fine for afpacket and pcap, for pf_ring we'd have to figure out what the various PF_RING licenses allow us to do, as well as how to correctly modprobe them. I do wonder, though, if just adding sudo:required to the test you're discussing might allow you to add it in as is.

[notti]

Hmm well when I looked around I found travis-ci/travis-ci#2291 which states that loading kernel modules is not supported. I have to admit I actually didn't try out loading kernel modules...

As for license - vanilla pf_ring is gplv2 for kernel and lgplv2 for userspace libraries. Only components we do not need, need a special license (pf_ring_zc, nprobe, n2disk). Additionally, pf_ring points to here for a golang userspace lib (https://github.com/ntop/PF_RING/blob/dev/userland/go/README) - so it should be possible to work something out

[notti]

ok installing pf_ring via dkms worked and modprobe didn't throw an error - tomorrow I'll try out if it actually worked

[notti]

Tried everything out and looks like loading kernel modules works perfectly - even self compiled ones.

[notti]

I had a look at your solution for stenographer and think this is not the best way for gopacket because testing sending out packets would be a bit more complicated or need the same capturing. Additionally, I think it would be better to have something, which can have both sides (sending with gopacket + receiving and comparing, or the other way around) directly inside go test with easily accessible methods.

So I had a look at tun/tap. I think this also represents the environment a bit better, since we basically emulate the network card with a file descriptor which can receive/send packets with read and write (basically our "virtual network card) and on the other end a network device, that looks like a real network device. With this it would be also possible to test without promisc mode, if needed.

Test setup can be seen at: https://github.com/notti/gopacket-taptest-poc https://travis-ci.com/notti/gopacket-taptest-poc

This testsetup

1. creates a tap device in tunAlloc which
2. creates a tap device with the given name (gopacket0)
3. disables ipv6 on the device (otherwise we would get spurios packets for dad and router solicitations - this could be changed to disable both and keep ipv6)
4. adds an ipv4 address and a netmask (not needed if we just test in promisc mode)
5. and finally brings up the device
6. spawns a goroutine, that listens on the fd for a packet, which it replies to immediately

• packet format on fd is a single packet per read/write
• if IFF_NO_PI is not set, the packet is preceded with 2 bytes flags and 2 bytes packet type

3. opens the the network device with pf_ring
4. sends a packet
5. receives a packet

Step 6 would be closing the file descriptor, after which the tap network device is removed. The devices live as long something is connected to them. Alternatively the tap network device can be made persistent with an additional ioctl before closing the fd.

Steps 4 and 5 are preceeded with TEST in the logoutput and step 2 with MONITOR

The travis part is also very simple:

wget http://apt-stable.ntop.org/`lsb_release -r | cut -f2`/all/apt-ntop-stable.deb
sudo dpkg -i apt-ntop-stable.deb
sudo apt-get update
sudo apt-get install linux-headers-`uname -r` pfring-dkms pfring libpcap-dev -y
sudo modprobe pf_ring
sudo modprobe tun

First two lines install the ntop repository, which has pf_ring debs. install linux-headers-uname -r pfring-dkms pfring installs pfring userspace library and the kernelmodule. Rest just loads the modules.

The test is executed by compiling it and then running it with sudo. This must be in two steps, since per default sudo cleans PATH and then go gets a bit upset...

.travis.gopacket_pfring.sh is not needed - this just overrides gopacket to pull request #553, since the ntop packages install version 7.2, which doesn't work without this.

Questions

• Does this look better/is this a viable way?
• If so, should I add a more generic tun/tap module to pcapgo and then use this to provide a small api for easier testing?

[gconnell]

tun/tap seems like another excellent way to go about this, so I'd say go for it!

Maybe we can add a gopacket/tuntap subpackage with the necessary stuff for easily creating tun/tap interfaces, then we can start writing some tests in other directories (/pcap, /afpacket, /pfring) to use it?

Obviously, tun/tap and anything that uses it will need a +linux build tag, so for tests in the /pcap directory we'll probably need to cordon those tests into separate files.

[notti]

Maybe we can add a gopacket/tuntap subpackage with the necessary stuff for easily creating tun/tap interfaces, then we can start writing some tests in other directories (/pcap, /afpacket, /pfring) to use it?

Yes, that would be my plan.

Obviously, tun/tap and anything that uses it will need a +linux build tag, so for tests in the /pcap directory we'll probably need to cordon those tests into separate files.

Well as far as I know, there is also tuntap for osx available, and a quick search also turns up something for windows. Since I have neither of those two available for coding looks like we have to start out with +linux and hope somebody else has a look at those two.

I'll close this for now, start coding/designing a generic tuntap interface and will make a pull request as soon as it's ready. When that one landed in gopacket, I'll try to come up with a testing framework.
Might take some time since cookie baking season started 🍪.

[gconnell]

Note: I accept cookies as well as pull requests.