container can't start - failed to move 'br0' to the container

[leeorr-skrumble]

Whenever I run lxc-start my containers fail to start. Running with a DEBUG log file I get this issue lxc_conf - conf.c:lxc_assign_network:3054 - failed to move 'br0' to the container : Invalid argument. In my google searches, I found that this used to happen due to an old kernel version, but I haven't found references to this issue since ~2012-2013.

There are also a couple of weird things about it:

1. the container will start if I set it to autostart, but if I then kill it and try to start it again it will fail.
2. Even weirder - this only happens on a specific server, even if I re-image it. If I follow the exact same steps on a virtualbox machine (same versions as well) it works perfectly. Since the machine where this happens is provided by an external source, I'm guessing this points to something in their initial configuration of the machine (before we get it) breaking this. I just don't have a clue what it could be.

Any ideas on what could be causing this? Or how it can be solved?

Here's a larger chunk of the log file:

xc-start 20160907105659.365 DEBUG    lxc_conf - conf.c:instantiate_veth:2615 - instantiated veth 'veth3X8DKP/vethY1HE9Y', index is '19'
      lxc-start 20160907105659.365 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for debian8
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.deny' set to 'a'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c *:* m'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'b *:* m'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 1:3 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 1:5 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 1:7 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 5:0 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 5:1 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 5:2 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 1:8 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 1:9 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 136:* rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 10:229 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 254:0 rm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 10:200 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 10:228 rwm'
      lxc-start 20160907105659.368 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1656 - cgroup 'devices.allow' set to 'c 10:232 rwm'
      lxc-start 20160907105659.368 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:1660 - cgroup has been setup
      lxc-start 20160907105659.421 ERROR    lxc_conf - conf.c:lxc_assign_network:3054 - failed to move 'br0' to the container : Invalid argument
      lxc-start 20160907105659.421 ERROR    lxc_start - start.c:lxc_spawn:1198 - failed to create the configured network
      lxc-start 20160907105659.554 ERROR    lxc_start - start.c:__lxc_start:1354 - failed to spawn 'debian8'

Here's the process I've gone through:

I'm running on an Ubuntu 16.04.1 host, freshly installed.

1. apt-get install lxc bridge-utils
   (lxc is 2.0.4)
2. set up a bridged interface (br0) by setting the following in /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
    address *IP*
    netmask *netmask*
    gateway *gateway*
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

3. run lxc-create -n debian8 -t debian -- -r jessie
4. edit /var/lib/lxc/debian8/config in the following manner:

• set the network link to br0: lxc.network.link = br0
• add: lxc.network.ipv4 = *container_ip/subnet* and lxc.network.ipv4.gateway = *gateway* for networking
• add auto-restart: lxc.start.auto = 1, lxc.start.delay = 5

5. edit /var/lib/lxc/debian8/rootfs/etc/network/interface so that eth0 is set to manual rather than dhcp.

Thanks!

[hallyn]

Please show your full container configuration, the contents of
/etc/lxc/default.conf, the host output of 'ifconfig -a' and
your lxc version.

[leeorr-skrumble]

I found the issue - it was actually related to a combintation of this systemd bug: systemd/systemd#3374 and udev rules supplied by our host provider that made ubuntu use the 14.04 style interface naming conventions. Whenever LXC tried to start a machine, udev would try to rename the VETH interface, which it failed cause LXC was using it, but still threw an error making lxc stop.

[stgraber]

Oh, that's a weird issue :)

So sounds like it's not LXC's fault and you found a way out of this problem, closing.