snabbswitch traffic crash

[ghost]

This issue is related to #560. However described is a method to reproduce it with Linux guest.

While sending traffic to a VM connected to snabbswitch it crashes with:

Get features 0x18428001
 VIRTIO_F_ANY_LAYOUT VIRTIO_NET_F_MQ VIRTIO_NET_F_CTRL_VQ VIRTIO_NET_F_MRG_RXBUF VIRTIO_RING_F_INDIRECT_DESC VIRTIO_NET_F_CSUM
lib/virtio/net_device.lua:404: attempt to index a nil value
stack traceback:
        core/main.lua:118: in function '__newindex'
        lib/virtio/net_device.lua:404: in function 'set_vring_num'
        apps/vhost/vhost_user.lua:216: in function 'method'
        apps/vhost/vhost_user.lua:142: in function 'process_qemu_requests'
        apps/vhost/vhost_user.lua:39: in function 'fn'
        core/timer.lua:33: in function 'call_timers'
        core/timer.lua:42: in function 'run_to_time'
        core/timer.lua:23: in function 'run'
        core/app.lua:235: in function 'main'
        program/snabbnfv/traffic/traffic.lua:85: in function 'traffic'
        program/snabbnfv/traffic/traffic.lua:61: in function 'run'
        program/snabbnfv/snabbnfv.lua:15: in function 'run'
        core/main.lua:56: in function <core/main.lua:32>
        [C]: in function 'xpcall'
        core/main.lua:125: in main chunk
        [C]: at 0x0044d9e0
        [C]: in function 'pcall'
        core/startup.lua:1: in main chunk
        [C]: in function 'require'
        [string "require "core.startup""]:1: in main chunk

To reproduce, checkout the Linux Ubuntu Trusty kernel:

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git
cd ubuntu-trusty
make defconfig

Enable the following in the `.config' file:

CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_BLK=y

Then compile with make bzImage -j8. The qemu command to run the test is:

taskset -c 3,11 \
qemu-system-x86_64 -m 1024 --enable-kvm -smp 2 \
        -drive file=ubuntu-trusty.img,if=virtio \
        -kernel bzImage \
        -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on \
        -numa node,memdev=mem \
        -chardev socket,id=char1,path=/tmp/vh0.sock,server \
        -netdev type=vhost-user,id=mynet1,chardev=char1 \
        -device virtio-net-pci,netdev=mynet1,addr=0x8,mac=00:16:3e:1a:b3:01 \
        -append "earlyprintk console=ttyS0 root=/dev/vda rootwait rw" -nographic

The snabbswitch ports file is:

return {
  { vlan = nil,
    mac_address = "00:16:3e:1a:b3:01",
    port_id = "0",
    ingress_filter = nil,
    gbps = nil,
    tunnel = nil
  },
}

The traffic is a TCP stream send from an external machine like that:

cat /dev/urandom | ssh root@100.0.0.1 "cat > /dev/null"

Follpwoing are the observations made till now:

1. When -smp 1 is ussed, the issue does not appear.
2. The issue is the same with vanilla kernel. Tested 3.13, 3.16, 3.18, 4.0
3. The analysis shows if the standard Ubuntu features are enabled, the issue does not appear:
   cat debian.master/config/config.common.ubuntu debian.master/config/amd64/config.common.amd64 debian.master/config/amd64/config.flavour.generic > .config
4. with some debug enabled it is seen that once the high traffic is started, the QEMU starts to send vhost_user: request set_vring_call 13. Which probably is a sign of the guest kernel trying to disable the interrupts. After sending this call several times in the end it sends:

vhost_user: request     set_vring_call  13
vhost_user: request     get_features    1
vhost_user: request     set_vring_num   8
lib/virtio/net_device.lua:404: attempt to index a nil value

And the crash occurs. get_features and set_vring_num are part of virtq initialization which probably means that the device is reset.

@lukego we are investigating the issue further and will post more info and eventually patches. Please assign this issue to me.

[lukego]

@virtualopensystems-nnikolaev Understood. This sounds related to issues that @mwiget has seen recently too. You will need to accept a Github Invitation that I sent before I can assign the issue to you.

Interesting development to see corporate-sponsored usernames on Github. I wonder if I could rename to coke-redbull-lukego and rake in some sponsorship bucks ;-)

[lukego]

@virtualopensystems-nnikolaev btw: you guys could consider creating a branch called e.g. vosys-nfv and registering it in src/doc/branches.md. Then you could collect fixes that you want to submit on that branch and PR them to master when you are ready to release them. This would give the vosys team proper credit for contributions because the vosys-nfv branch would be visible in both PRs and git merge commits. It would also allow Snabb Switch users to discover your branch via the SnabbCo repo.

I definitely see it as a positive for contributors to get proper credit (both as individuals and companies) so that positive contributions to the upstream repo will be a valuable marketing tool.

[lukego]

@virtualopensystems-nnikolaev This is a very good Issue you have written :). I am also interested in fixing this. Can I help somehow?

[ghost]

@lukego, first we need to understand what it means for the kernel to send those vhost_user: request set_vring_call 13. It probably tries to suppress interrupts although there is also other mechanism to do that (see VRING_F_NO_INTERRUPT in VirtioVirtq:signal_used()).

Then - we should be able to detect that a re-initialization is happening and to shutdown our virtqs properly.

I also very interesting that -smp 1 does not trigger it, might have to do something with using barriers etc.
All in all - it is about debugging and getting more info, maybe more on the QEMU/Linux kernel side.

[lukego]

Here is a relevant qemu-devel thread suggesting that the intended vhost-user behavior in QEMU is not fully locked down with respect to guest reboot vs. guest driver unload/reload. Could be related to the issue we are seeing.

@eugeneia is working on adding a CI test case where the guest reloads the virtio_net kernel module to see if that triggers this problem or a related one. (@virtualopensystems-nnikolaev do you have any tips on bootstrapping that kernel/guest? do we need to add an additional make modules_install compared with the old instructions from bench_env README? I suppose @eugeneia can shout directly if he needs tips.)

[eugeneia]

I built virtio_net as a module and a kernel without it being built in (CONFIG_VIRTIO_NET=m), then I installed that module into a copy of the test_env qemu image by hand (I know that's bad workflow, how do I do it right? E.g. as a modification to the bench_env/README). I ran luke's test case which consists of:

ifconfig eth0 down
rmmod virtio_net
# Never got to the steps below
modprobe virtio_net
ifconfig eth0 up

and got the following:

lib/virtio/net_device.lua:357: mapping to host address failedcdata<void *>: 0xffff88001e80a430
stack traceback:
    core/main.lua:116: in function <core/main.lua:114>
    [C]: in function 'error'
    lib/virtio/net_device.lua:357: in function 'map_from_guest'
    lib/virtio/net_device.lua:129: in function 'packet_start'
    lib/virtio/virtq.lua:67: in function 'get_buffers'
    lib/virtio/net_device.lua:122: in function 'receive_packets_from_vm'
    lib/virtio/net_device.lua:108: in function 'poll_vring_receive'
    apps/vhost/vhost_user.lua:76: in function 'method'
    core/app.lua:76: in function 'with_restart'
    core/app.lua:283: in function 'breathe'
    core/app.lua:237: in function 'main'
    program/snabbnfv/traffic/traffic.lua:127: in function 'bench'
    program/snabbnfv/traffic/traffic.lua:59: in function 'run'
    program/snabbnfv/snabbnfv.lua:15: in function 'run'
    core/main.lua:56: in function <core/main.lua:32>
    [C]: in function 'xpcall'
    core/main.lua:121: in main chunk
    [C]: at 0x0044e670
    [C]: in function 'pcall'
    core/startup.lua:1: in main chunk
    [C]: in function 'require'
    [string "require "core.startup""]:1: in main chunk

[lukego]

@eugeneia OK. Looks like an unrelated bug. I think we should cover this in the CI tests. I think we should also Dockerize the bootstrapping of kernels and images: I don't think we should ever be doing this stuff by hand anymore. You interested in handling that?

There is a Docker feature called "volumes" that seems relevant: one container could build the artifacts (kernel and guest image) and then export them as a "volume" that another container (SnabbBot) could access. This should make it possible to mix-and-match the same SnabbBot with many different kernel/image combos. (Could also be that this is the wrong solution.)

[eugeneia]

@lukego Can we move the docker discussion into a separate issue?

[ghost]

Tracing down the issue, we found out that under high load the virtio-net driver in the guest is generating a lot of interrupt mask/unmask events to the virtio-net-pci device in QEMU. This results in QEMU resetting the virtio-net-pci device after ~1 minute of high load (tested with VM2VM example).
The generation of these events also slows down the throughput.

This behavior is not seen provided that the following .config options are set:

CONFIG_X86_X2APIC=y
CONFIG_KVM_GUEST=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y

It seems that the critical part is to enable X2APIC support in the guest, which forwards the interrupts to the KVM instead of using the QEMU emulated interrupt controller.

[eugeneia]

@nnikolaev-virtualopensystems Does this mean this issue is “fixed”? Can we do anything further than ensuring the kernel was compiled with the right parameters?