Qemu-emulated NICs don't work for inter-VM traffic

[marmarek]

Reported by joanna on 2 Jan 2013 11:25 UTC
Inter-appvm traffic doesn't work when one of the AppVMs is an HVM and uses qemu-emulated networking (this is that case e.g. when one have some older Linux distro as one of the HVM).

Steps to reproduce:

1. Install an older Linux distro as an HVM (I installed Ubuntu 10.04 i386)
2. Start some normal AppVM
3. Setup inter-appvm networking between the AppVM and the HVM, according to the instructions here:

https://wiki.qubes-os.org/trac/wiki/QubesFirewall

4. Try pinging the HVM from AppVM -- this should work
5. Try ssh (or use any other tcp/udp connection, even netcat) to the HVM -- this would not work, surprisingly.

Note, if, instead of an older Ubuntu I use the very recent Ubuntu 12.10 (that has xen pv drivers builtin), the above setup works fine.

Also, note that the networking in the HVM (the old ubuntu) actually works fine -- I can e.g. browse the web fine. So, the traffic that comes from the outside world, and which arrive to the HVM interfaces is processed fine. But the traffic (other than ICMP!) that arrives from other AppVMs (or even from the FirewallVM) is... discarded. Specifically, when I run tcpdump in the HVM, I can see the incoming SYN packates (e.g. to SSH port) but I see not SYN|ACK nor RST packets being generated in response. It seems like the HVM's kernel is discarding the incoming packets before sending them down the TCP stack (but, again, the ICMP request packets are processes correctly, and ICMP responses are generated).

Quite a strange case...

Migrated-From: https://wiki.qubes-os.org/ticket/700

[marmarek]

Modified by joanna on 2 Jan 2013 11:33 UTC

[marmarek]

Modified by joanna on 2 Jan 2013 11:33 UTC

[marmarek]

Comment by marmarek on 8 Jan 2013 02:03 UTC
The problem looks to be not working tx-checksumming offload in xen-netfront. When I turn it off in source VM, tcp connections start working:

ethtool -K eth0 tx off

It have worked for outside traffic because real network device (in netvm) have working this offload, so calculated checksum correctly. ICMP was working most likely because if was calculated by the kernel, not left for the offload.

[marmarek]

Comment by marmarek on 8 Jan 2013 02:03 UTC
The problem looks to be not working tx-checksumming offload in xen-netfront. When I turn it off in source VM, tcp connections start working:

ethtool -K eth0 tx off

It have worked for outside traffic because real network device (in netvm) have working this offload, so calculated checksum correctly. ICMP was working most likely because if was calculated by the kernel, not left for the offload.

[marmarek]

Comment by marmarek on 8 Jan 2013 02:04 UTC
http://git.qubes-os.org/?p=marmarek/core.git;a=commit;h=84375c356752073231ce1dab36e5cd45027bb600

[marmarek]

Comment by marmarek on 8 Jan 2013 02:04 UTC
http://git.qubes-os.org/?p=marmarek/core.git;a=commit;h=84375c356752073231ce1dab36e5cd45027bb600