could not get private network working

[ebayer]

On Opensuse Tumbleweed, I could not get private networking to work with a CentOS7 container. Here are the relevant info (I think):

$ lsb_release -a
LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID: openSUSE
Description:    openSUSE Tumbleweed
Release:        20190305
Codename:       n/a

$ sudo lxc-info --version
2.0.9

$ vagrant --version
Vagrant 2.2.2

$ vagrant plugin list
vagrant-cachier (1.2.1, global)
vagrant-libvirt (0.0.45, global)
vagrant-lxc (1.4.3, global)

Using the following Vagrantfile and starting the container with debug:

Vagrant.configure("2") do |config|
  config.vm.define "test1", primary: true do |test1|
    test1.vm.hostname = "test1"
    test1.vm.network "private_network", ip: "192.168.2.100", lxc__bridge_name: 'vlxcbr1'
  end
end

$ REDIR_LOG=1 LXC_START_LOG_FILE=/tmp/lxc-start.log VAGRANT_LOG=debug vagrant up --provider=lxc test1
 INFO global: Vagrant version: 2.2.2
 INFO global: Ruby version: 2.6.1
 INFO global: RubyGems version: 3.0.1
 INFO global: VAGRANT_LOG="debug"
...
 INFO interface: output: Setting up private networks...
 INFO interface: output: ==> test1: Setting up private networks...
==> test1: Setting up private networks...
 INFO driver: Configuring network interface for oyas_test1_1552473237752_9965 using 192.168.2.100 and bridge vlxcbr1
 INFO driver: Checking whether bridge vlxcbr1 exists
 INFO driver: Creating the bridge vlxcbr1
 INFO subprocess: Starting process: ["/usr/bin/sudo", "/usr/local/bin/vagrant-lxc-wrapper", "brctl", "addbr", "vlxcbr1"]
 INFO subprocess: Vagrant not running in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO driver: Checking whether the bridge vlxcbr1 has an IP
 INFO driver: Adding 192.168.2.254 to the bridge vlxcbr1
 INFO subprocess: Starting process: ["/usr/bin/sudo", "/usr/local/bin/vagrant-lxc-wrapper", "ip", "addr", "add", "192.168.2.254/24", "dev", "vlxcbr1"]
 INFO subprocess: Vagrant not running in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO subprocess: Starting process: ["/usr/bin/sudo", "/usr/local/bin/vagrant-lxc-wrapper", "ip", "link", "set", "vlxcbr1", "up"]
 INFO subprocess: Vagrant not running in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO subprocess: Starting process: ["/usr/bin/sudo", "/usr/local/bin/vagrant-lxc-wrapper", "/home/ebayer/.vagrant.d/gems/2.6.1/gems/vagrant-lxc-1.4.3/scripts/pipework", "vlxcbr1", "oyas_test1_1552473237752_9965", "192.168.2.100/24"]
 INFO subprocess: Vagrant not running in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stderr: /home/ebayer/.vagrant.d/gems/2.6.1/gems/vagrant-lxc-1.4.3/scripts/pipework: line 393: Address:: command not found
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 31999
DEBUG subprocess: Exit status: 0
 INFO warden: Calling OUT action: #<Vagrant::LXC::Action::PrivateNetworks:0x0000000002638e98>

I can see that new bridge is created and got the default ip address, but state is down:

$ ip a
7: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 scope global lxcbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe00:0/64 scope link 
       valid_lft forever preferred_lft forever
120: vlxcbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:58:14:0e:64:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.254/24 scope global vlxcbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::5022:71ff:fe98:a96e/64 scope link 
       valid_lft forever preferred_lft forever
122: veth1pl11149@if121: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master vlxcbr1 state LOWERLAYERDOWN group default qlen 1000
    link/ether 52:58:14:0e:64:cb brd ff:ff:ff:ff:ff:ff link-netnsid 0

$ sudo brctl show
vlxcbr1         8000.5258140e64cb       no              veth1pl11149

And interface inside the container is created but without an ip address:

$ vagrant ssh
[vagrant@test1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
118: eth0@if119: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:ff:a2:fc:3f:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.3.75/24 brd 10.0.3.255 scope global dynamic eth0
       valid_lft 3544sec preferred_lft 3544sec
    inet6 fe80::cff:a2ff:fefc:3fd2/64 scope link 
       valid_lft forever preferred_lft forever
121: eth1@if122: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ba:53:fd:82:03:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0

If I ssh into container and set the interface to up and give an ip address, it works:

$ vagrant ssh
[vagrant@test1 ~]$ sudo ip l set up dev eth1
[vagrant@test1 ~]$ sudo ip addr add 192.168.2.100/24 dev eth1
[vagrant@test1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
118: eth0@if119: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0e:ff:a2:fc:3f:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.3.75/24 brd 10.0.3.255 scope global dynamic eth0
       valid_lft 2800sec preferred_lft 2800sec
    inet6 fe80::cff:a2ff:fefc:3fd2/64 scope link 
       valid_lft forever preferred_lft forever
121: eth1@if122: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ba:53:fd:82:03:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.2.100/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::b853:fdff:fe82:316/64 scope link 
       valid_lft forever preferred_lft forever

And bridge and new interface status are fixed on the host after that:

$ ip a
120: vlxcbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:58:14:0e:64:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.254/24 scope global vlxcbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::5022:71ff:fe98:a96e/64 scope link 
       valid_lft forever preferred_lft forever
122: veth1pl11149@if121: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vlxcbr1 state UP group default qlen 1000
    link/ether 52:58:14:0e:64:cb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::5058:14ff:fe0e:64cb/64 scope link 
       valid_lft forever preferred_lft forever

How can I further debug this? Is there some pre-start tasks that I should do on the host?

My lxc and lxc-net configs are like this:

$ sudo grep -i bridge /etc/default/lxc
USE_LXC_BRIDGE="true"  # overridden in lxc-net

$ sudo cat /etc/lxc/default.conf
#lxc.network.type = empty
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up

[00raymond00]

I just had this same issue. It turns out that pipework does not correctly obtain the IP address to set up if you have ipcalc installed on your local UBUNTU 16.04 box. So just remove the ipcalc package on your local box and you're good to go.

A more detailed explanation...

The ipcalc portion of the pipework code is broken. ipcalc on UBUNTU 16.04 has different options than the ones the pipework script uses. In pipework, it uses the -b flag for ipcalc to obtain the broadcast address. ipcalc on UBUNTU doesn't have the -b flag.

I turned on debugging for both vagrant and the pipework script.

With ipcalc installed I saw the following output when bringing up a Vagrant node:

DEBUG subprocess: stderr: + [ bridge = phys ]
+ [ bridge = ipoib ]
+ [ bridge = dummy ]
+ [ bridge = route ]
+ ip link set veth1pg150166 netns 150166
DEBUG subprocess: stderr: + ip netns exec 150166 ip link set veth1pg150166 name eth1
DEBUG subprocess: stderr: + [ ]
+ installed ipcalc
+ command -v ipcalc
DEBUG subprocess: stderr: + ipcalc -b 192.168.2.2/24
DEBUG subprocess: stderr: + eval Address: 192.168.2.2 Netmask: 255.255.255.0 = 24 Wildcard: 0.0.0.255 => Network: 192.168.2.0/24 HostMin: 192.168.2.1 HostMax: 192.168.2.254 Broadcast: 192.168.2.255 Hosts/Net: 254 Class C, Private Internet
DEBUG subprocess: stderr: + Address: 192.168.2.2 Netmask: 255.255.255.0 = 24 Wildcard: 0.0.0.255 = 192.168.2.0/24 HostMin: 192.168.2.1 HostMax: 192.168.2.254 Broadcast: 192.168.2.255 Hosts/Net: 254 Class C, Private Internet
/data1/user.name/.vagrant.d/gems/2.4.9/gems/vagrant-lxc-1.4.3/scripts/pipework: 1: eval: Address:: not found
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0

With ipcalc removed from my local box and bring up a Vagrant node:

DEBUG subprocess: stderr: + [ bridge = phys ]
+ [ bridge = ipoib ]
+ [ bridge = dummy ]
+ [ bridge = route ]
+ ip link set veth1pg157647 netns 157647
DEBUG subprocess: stderr: + ip netns exec 157647 ip link set veth1pg157647 name eth1
DEBUG subprocess: stderr: + [ ]
+ installed ipcalc
+ command -v ipcalc
+ ip netns exec 157647 ip addr add 192.168.2.2/24 dev eth1
DEBUG subprocess: stderr: + [ ]
+ ip netns exec 157647 ip link set eth1 up
DEBUG subprocess: stderr: + [ ]
+ installed arping
+ command -v arping
DEBUG subprocess: stderr: + echo 192.168.2.2/24
DEBUG subprocess: stderr: + cut -d/ -f1
DEBUG subprocess: stderr: + IPADDR=192.168.2.2
+ ip netns exec 157647 arping -c 1 -A -I eth1 192.168.2.2
DEBUG subprocess: stderr: + rm -f /var/run/netns/157647
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 31999
DEBUG subprocess: Exit status: 0

[fgrehm]

Hey, sorry for the silence here but this project is looking for maintainers 😅

As per #499, I've added the ignored label and will close this issue. Thanks for the interest in the project and LMK if you want to step up and take ownership of this project on that other issue 👋