Fix Debian base boxes networking issue

[fgrehm]

Just built the boxes using the new scripts and while trying them on a Raring host I noticed that the containers boots but do not get an ip, an attempt to bring the a wheezy container up by hand with sudo lxc-start -n $(cat .vagrant/machines/default/lxc/id) yields:

NIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
mount: cannot remount block device tmpfs read-write, is write-protected
mount: cannot remount block device tmpfs read-write, is write-protected
mount: cannot remount block device proc read-write, is write-protected
mount: cannot remount block device sysfs read-write, is write-protected
mount: cannot remount block device tmpfs read-write, is write-protected
mount: cannot remount block device devpts read-write, is write-protected
Mount point '/dev/lxc/console' does not exist. Skipping mount. ... (warning).
Mount point '/dev/lxc/tty1' does not exist. Skipping mount. ... (warning).
Mount point '/dev/lxc/tty2' does not exist. Skipping mount. ... (warning).
Mount point '/dev/lxc/tty3' does not exist. Skipping mount. ... (warning).
Mount point '/dev/lxc/tty4' does not exist. Skipping mount. ... (warning).
Mount point '/dev/ptmx' does not exist. Skipping mount. ... (warning).
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux 2.20.1
done.
Mounting local filesystems...done.
/etc/init.d/mountall.sh: 59: kill: Illegal number: 2 1
Activating swapfile swap...done.
Setting kernel variables ...done.
Configuring network interfaces...Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/00:16:3e:d5:bf:c6
Sending on   LPF/eth0/00:16:3e:d5:bf:c6
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
5 bad udp checksums in 5 packets
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
No DHCPOFFERS received.
No working leases in persistent database - sleeping.
done.
INIT: Entering runlevel: 3
Using makefile-style concurrent boot in runlevel 3.
Starting OpenBSD Secure Shell server: sshd.

Dunno what to do in order to fix it right now.

[fcharlier]

This is due to dhclient checking the checksum of udp packets it receives, but checksums are not computed if the packets don't leave through a physical interface.

You can add an iptables rule to force adding checksums (replace your network range eventually) : post-up iptables -A POSTROUTING -t mangle -p udp --dport bootpc -s 172.20.0.1/24 -j CHECKSUM --checksum-fill

[fgrehm]

@fcharlier tks for the input :) TBH this is pretty weird to me as it used to work in the past and I thought I had moved all the custom stuff from the "old" templates into the new script. looks like I might need to look at the git logs again to find out if I missed out on something.

[stucki]

Seems like an LXC issue to me. Here is a patch for the lxc-net init script on Ubuntu hosts:

diff --git a/init/lxc-net.conf b/init/lxc-net.conf
index 71f3e7d..b0ebf03 100644
--- a/init/lxc-net.conf
+++ b/init/lxc-net.conf
@@ -42,6 +42,9 @@ pre-start script
        iptables -t nat -A POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE
        dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=${varrun}/dnsmasq.pid --conf-file= --listen-address ${LXC_ADDR} --dhcp-range ${LXC_DHCP_RANGE} --dhcp-lease-max=${LXC_DHCP_MAX} --dhcp-n
        touch ${varrun}/network_up
+
+       # Fix for https://github.com/fgrehm/vagrant-lxc/issues/153
+       iptables -t mangle -A POSTROUTING -p udp --dport bootpc -s ${LXC_NETWORK} -j CHECKSUM --checksum-fill
 end script

 post-stop script
@@ -58,4 +61,7 @@ post-stop script
                brctl delbr ${LXC_BRIDGE}
        fi
        rm -f ${varrun}/network_up
+
+       # Fix for https://github.com/fgrehm/vagrant-lxc/issues/153
+       iptables -t mangle -D POSTROUTING -p udp --dport bootpc -s ${LXC_NETWORK} -j CHECKSUM --checksum-fill || true
 end script

[tdhite]

Agreed on the patch above. Had to do the same thing.

[fgrehm]

Thanks guys! I'll give it a try later on and will add an entry to our troubleshooting section of the Wiki if it works for me as well :)

[fgrehm]

Oh folks, seems like this should have more attention after a long debugging session over twitter with @rcarmo today as he experienced that with an Ubuntu guest as well if I'm not wrong.

I'm willing to look into fixing this but I have no idea how to detect that issue prior to booting the container so we can either fix things automatically or let the users know that things are misconfigured and let them choose whether they want to do the changes or not.

Anyone has any thoughts?

(/cc @leorochael as you seem to be interested on this as well :)

[rcarmo]

I'd say you should check for the existence of an iptables rule that does the checksumming and add it if missing. Period :)

On Oct 16, 2013, at 03:50 , Fabio Rehm notifications@github.com wrote:

Oh folks, seems like this should have more attention after a long debugging session over twitter with @rcarmo today as he experienced that with an Ubuntu guest as well if I'm not wrong.

I'm willing to look into fixing this but I have no idea how to detect that issue prior to booting the container so we can either fix things automatically or let the users know that things are misconfigured and let them choose whether they want to do the changes or not.

Anyone has any thoughts?

(/cc @leorochael as you seem to be interested on this as well :)

—
Reply to this email directly or view it on GitHub.

[rcarmo]

Also, I've just upgraded my 13.04/lxc 0.9.0 VM to saucy/lxc 1.0.0. Adding the checksumming made things work as well.

[TiagoTT]

I have hit the same issue and found another solution on Xen wiki: http://wiki.xen.org/wiki/Xen_Networking

Instead of having iptables calculate the checksum, the interface can be configured to disable offload checksum and thus make the OS network stack do all the work without relying on hardware (which isn't there, because its a bridge).

ethtool -K br0 tx off

Here I present the packet capture before and after the configuration change:
https://gist.github.com/TiagoTT/7011886

Notice the problematic [bad udp cksum 0x78fc -> 0xf8aa!] on DHCP Reply of the first capture, which becomes [udp sum ok] on the DHCP Reply of the second capture.

This works just like the iptables rule, but I find it cleaner.

[rcarmo]

Yeah. I tested this with Tiago and it works if you add this just after brctl addbr:

ethtool -K ${LXC_BRIDGE} tx off

This can also be put inside vagrant-lxc, although you'll need to have ethtool installed (which it isn't by default, at least not on my barebones 13.04 and 13.10 VMs)

R.

On Oct 16, 2013, at 18:49 , Tiago Teresa Teodósio notifications@github.com wrote:

I have hit the same issue and found another solution on Xen wiki: http://wiki.xen.org/wiki/Xen_Networking

Instead of having iptables calculate the checksum, the interface can be configured to disable offload checksum and thus make the OS network stack do all the work without relying on hardware (which isn't there, because its a bridge).

ethtool -K br0 tx off

Here I present the packet capture before and after the configuration change:
https://gist.github.com/TiagoTT/7011886

Notice the problematic [bad udp cksum 0x78fc -> 0xf8aa!] on DHCP Reply of the first capture, which becomes [udp sum ok] on the DHCP Reply of the second capture.

This works just like the iptables rule, but I find it cleaner.

—
Reply to this email directly or view it on GitHub.

[fgrehm]

Tks guys!

Just to confirm, will this have any sort of side effect if we end up doing automatically? Should we ask for confirmation prior to running the command?

I just want to make sure we don't mess up with users networks by doing this automatically :P

[fgrehm]

As I said on other issues, my networking skillz are still close to none at this moment so how can we identify the issue in advance prior to running the command? I mean, which commands do I need to run in order to identify the problem?

[rcarmo]

I can't see what sort of side effects it would bring, since I have Ubuntu and Debian containers running with this set now.

What this does is disable the off-load of packet checksums to a hardware interface, which seems to have become the default ever since Ethernet cards became smart enough to checksum packets on their own.

This was done to save CPU time, but since we're not using a hardware interface in lxcbr0, there's no hardware to checksum the packets, and hence the host kernel seems to be sending them without checksums at all (they're supposed to be filled in inline by the hardware). But the guests consider those invalid packets.

I'm at a loss to explain if this is part of an LXC tweak, a kernel tweak, or some mis-placed default, but that's what's happening now. All that will happen is that SSHing to a container will take an extra, negligible amount of CPU (heck, we're encrypting the connection inside the machine already...).

R.

[fgrehm]

Tks for the information everyone but I'm still not sure how / when I'll get to fix this. Tonight I'll try to sum everything up into a single issue along with the proposed fixes and will add a BIG note to the README pointing users over there.
If someone is able to come up with a PR I'd be more than happy to review it and merge it in :D

[stucki]

While we have two suggested solutions now, I still wonder how this problem got introduced. Is it caused by recent kernel changes? LXC changes?

[rcarmo]

Our money's actually on the kernel. The guy who wrote ethtool seems to be related to the bridging module. But it might also be a change in LXC init somewhere.

[TiagoTT]

My analysis was not entirely correct. Please do not run ethtool -K ${LXC_BRIDGE} tx off in the init scripts.
Also the issue seems to be fixed with latest Ubuntu 13.10 update.

Running ethtool -K vifX.Y tx off for each virtual interface also fixes the problem.

[fgrehm]

Quick update: I was doing some testing on a clean Ubuntu 13.04 installation on DigitalOcean and after rebuilding the base wheezy box I was able to connect to it without issues.

[FrancoTampieri]

@fgrehm
An update, I tried to copy a container created without problems with vagrant on Ubuntu 13.04 and apart from some initial problem that has not kept some file permissions, it works normally without the patch.

The problem is only during creation of containers.

As soon as possible I'll try the patch that you have shown me.

[FrancoTampieri]

@fgrehm
I comfirm that i work without problem on ubuntu 13.04 since saturday, I have problem only with 13.10 release.

[FrancoTampieri]

@fgrehm
I fall back in lxc 1.0 from the stadard repository of ubuntu and applied the patch of #153 (comment) and all works.

in this w.e. I try again with the last stable release of lxc from the ubuntu team.

[rcarmo]

I had the occasion to set up another 13.10 container box. using ethtool was the only way to get it working after a dist-upgrade.

[eswierk]

The actual bug is in the ISC DHCP client, which is supposed to honor a flag (TP_STATUS_CSUMNOTREADY) indicating that the UDP checksum was never computed and should be ignored.

Various distributions are adding a patch to the DHCP client package that fixes this. For example:

https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/930962

Unfortunately the ethtool or iptables workaround will be needed for any guests with the buggy DHCP client.

[rcarmo]

Guests that previously worked fine started misbehaving after upgrading the host, so that's not the only factor.

On 29 Oct 2013, at 23:09 , Ed Swierk notifications@github.com wrote:

The actual bug is in the ISC DHCP client, which is supposed to honor a flag (TP_STATUS_CSUMNOTREADY) indicating that the UDP checksum was never computed and should be ignored.

Various distributions are adding a patch to the DHCP client package that fixes this. For example:

https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/930962

Unfortunately the ethtool or iptables workaround will be needed for any guests with the buggy DHCP client.

—
Reply to this email directly or view it on GitHub.

[eswierk]

Correct, the ISC DHCP client bug is not the only factor. Until recently, the veth driver didn't claim to support checksum offload and other features. Now that it does (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=8093315a91340bca52549044975d8c7f673b28a1), a packet traverses the entire path from LXC host to guest, through bridges and veths, without the checksum getting computed. Any client that doesn't notice this, and assumes the checksum is valid, would break.

[fgrehm]

Thanks for all the information folks. I still didn't have the time to dig into this but I ran into it while doing some testing with other stuff.

So I'd love to have vagrant-lxc erroring out / warning users in case we are able to detect the issue on the host. I personally don't like the idea of automatically running arbitrary commands without need (like on my laptop that doesn't have a need for it) or without letting the user know what's going on.

If someone could please point me to the buggy package version and / or a command that we can use to detect if the problem is in place I promise I'll implement it on the plugin pretty quick :D

[rcarmo]

Maybe checking for a specific kernel version is feasible. @eswierk, can you track that commit you pointed to into a release tag? I know Ubuntu tweaks their stuff, but...

[eswierk]

The minimal workaround is to disable TX checksum offload on the LXC's veth device by running ethtool -k vethXXX tx off. That command is effective only if the veth device advertises TX checksum offload support, which is the exact condition that triggers the ISC DHCP bug. So you might as well just do it--there's little value in going to the trouble of checking the kernel version first.

It would be more useful to check the version of the ISC DHCP client in the container, and apply the workaround only if it's an old version that doesn't deal with the missing checksum situation. But that check would depend on the exact distribution you've installed in the container. For Ubuntu you could use the bug fix release details from https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/930962.

[KampfCaspar]

On (K)Ubuntu 13.04 host with lxc 0.9.0-0ubuntu3.7, disabling checksum offloading seems to solve all networking issues on all boxes I tried.

On a 13.10 host with lxc 1.0.0alpha1-0ubuntu11 (as well as 1.0.0alpha2+master20131019-0306-0ubuntu1ppa1~saucy1 from ubuntu-lxc/stable), however, the exact same saucy64 box does not work. Sniffing the lxcbr0 interface, I get NO dhcp request, only icmpv6 router solicitations. After lxc-attaching and issuing 'ifup eth0' as well as '/usr/sbin/sshd' it continues (and dhcp packets are seen on lxcbr0)

[pleddy]

Just wanted to add two cents, been stuck on this for 24 hours. Just reiterating. Seems solved now, though I am only testing with basic lxc commands, not vagrant-lxc. Can we agree that using base lxc makes more sense for debugging?

root@my-macbook-air:~# lsb_release -a
Description: Ubuntu 13.10
Release: 13.10
Codename: saucy

Only this worked for me:

sudo iptables -t mangle -A POSTROUTING -o lxcbr0 -p udp --dport bootpc -j CHECKSUM --checksum-fill

I tried ethtool solution but did not work.

Once the iptables rule is added to host system -- note, people did not say above where to add the rule clearly -- the client system was able to fetch an IP from dnsmasq. Otherwise, I saw "bad udp cksum" messages in output of "tcpdump -vvv -i lxcbr0".

[fgrehm]

So... it has been 2 months since this issue had any activity and I wasn't able to reproduce it on Raring neither on Saucy. Besides that, I did some testing and the new Debian boxes built with the changes on GH-245 are working fine for me as well.

We already have a note on the readme pointing users to this issue for 4 months I'd expect more reports if it was an issue affecting lots of people.

If someone is able to hook me up with a VBox machine that consistently reproduces the bug or is able put up a PR that detects and / or fixes the problems I'll be more than happy to review / bring it in, but for now I'm just going to close the issue :-)

[fgrehm]

Folks, for whatever reason I ended up hitting this issue again on a Ubuntu Saucy VBox VM with a Debian Sid box. It's pretty weird that it doesn't happen on my laptop using Saucy as well o_O

Anyways, the easiest way I could work around it without changing anything on the plugin was to make use of the lxc.network.script.up config.

Basically I created /sbin/ethtool-fix with:

#!/bin/bash
/sbin/ethtool -K lxcbr0 tx off

And added the following to my Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.provider :lxc do |lxc|
    lxc.customize 'network.script.up', '/sbin/ethtool-fix'
  end
end

If you want that behavior for all of your vagrant-lxc containers, you can add that line above to your ~/.vagrant.d/Vagrantfile so that the project's specific Vagrantfile is not "polluted" with host's configuration. I'll add this information to the new docs website that will come out with 1.0 along with the other solutions listed here.