Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot launch new container after switching to lxdbr0 (ubunut 16.04 upgrade) #1857

Closed
alci63 opened this issue Apr 6, 2016 · 15 comments
Closed

Comments

@alci63
Copy link

@alci63 alci63 commented Apr 6, 2016

On Ubuntu 16.04 after latest upgrade, I had a message saying lxdbr0 supersedes lxcbr0. lxc1 package was removable and I purged it. Now, I can't launch a container anymore (be it new or existing).

Issue description

~$ lxc launch 16.04.b spip
Creating spip
Starting spip
error: Error calling 'lxd forkstart spip /var/lib/lxd/containers /var/log/lxd/spip/lxc.conf': err='exit status 1'
Try lxc info --show-log spip for more info
~$ lxc info --show-log spip | grep ERROR
lxc 20160406145009.135 ERROR lxc_conf - conf.c:instantiate_veth:2593 - failed to attach 'veth62XU1P' to the bridge 'lxdbr0': Operation not permitted
lxc 20160406145009.156 ERROR lxc_conf - conf.c:lxc_create_network:2870 - failed to create netdev
lxc 20160406145009.156 ERROR lxc_start - start.c:lxc_spawn:1066 - failed to create the network
lxc 20160406145009.156 ERROR lxc_start - start.c:__lxc_start:1329 - failed to spawn 'spip'

@niko-daniel

This comment has been minimized.

Copy link

@niko-daniel niko-daniel commented Apr 6, 2016

hi sir.
can you attach informations from

sudo service lxc-net status

and

sudo service lxd-bridge status

thanks

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Apr 6, 2016

We'd also need:

dpkg -l | grep lxd

The error means that lxdbr0 isn't running on your system. That could be because of some missing dependencies which I fixed late yesterday, if not, we need to figure out why it failed to start.

@alci63

This comment has been minimized.

Copy link
Author

@alci63 alci63 commented Apr 6, 2016

dpkg -l | grep lxd
ii  lxd                                                  2.0.0~rc8-0ubuntu5                            amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                                           2.0.0~rc8-0ubuntu5                            amd64        Container hypervisor based on LXC - client

franck@franck-ThinkPad-T430s:~$ sudo service lxd-bridge status
● lxd-bridge.service - LXD - network bridge
   Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
   Active: active (exited) since mer. 2016-04-06 12:55:06 CEST; 3h 18min ago
     Docs: man:lxd(1)
  Process: 22255 ExecStart=/usr/lib/lxd/lxd-bridge.start (code=exited, status=0/SUCCESS)
 Main PID: 22255 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/lxd-bridge.service

avril 06 12:55:06 franck-ThinkPad-T430s systemd[1]: Starting LXD - network bridge...
avril 06 12:55:06 franck-ThinkPad-T430s lxd-bridge.start[22255]: dnsmasq: ne peut ouvrir ou créer le fichiers de baux /var/lib/lxd-bridge//dnsmasq.lxdbr0.leases : Permiss
avril 06 12:55:06 franck-ThinkPad-T430s dnsmasq[22292]: ne peut ouvrir ou créer le fichiers de baux /var/lib/lxd-bridge//dnsmasq.lxdbr0.leases : Permission non accordée
avril 06 12:55:06 franck-ThinkPad-T430s dnsmasq[22292]: IMPOSSIBLE de démarrer
avril 06 12:55:06 franck-ThinkPad-T430s lxd-bridge.start[22255]: Failed to setup lxd-bridge.
avril 06 12:55:06 franck-ThinkPad-T430s systemd[1]: Started LXD - network bridge.

franck@franck-ThinkPad-T430s:~$ sudo service lxc-net status
● lxc-net.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

avril 06 12:11:21 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPACK(lxcbr0) 10.0.1.116 00:16:3e:5d:d6:59 lxd-digdash
avril 06 12:14:57 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPREQUEST(lxcbr0) 10.0.1.201 00:16:3e:78:39:ea
avril 06 12:14:57 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPACK(lxcbr0) 10.0.1.201 00:16:3e:78:39:ea lxd-spip
avril 06 12:33:54 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPREQUEST(lxcbr0) 10.0.1.116 00:16:3e:5d:d6:59
avril 06 12:33:54 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPACK(lxcbr0) 10.0.1.116 00:16:3e:5d:d6:59 lxd-digdash
avril 06 12:36:16 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPREQUEST(lxcbr0) 10.0.1.201 00:16:3e:78:39:ea
avril 06 12:36:16 franck-ThinkPad-T430s dnsmasq-dhcp[3787]: DHCPACK(lxcbr0) 10.0.1.201 00:16:3e:78:39:ea lxd-spip
avril 06 12:52:35 franck-ThinkPad-T430s systemd[1]: Stopping LXC network bridge setup...
avril 06 12:52:35 franck-ThinkPad-T430s systemd[1]: Stopped LXC network bridge setup.
avril 06 12:52:35 franck-ThinkPad-T430s systemd[1]: Stopped LXC network bridge setup.

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Apr 6, 2016

Can you run:

chown -R lxd:lxd /var/lib/lxd-bridge
systemctl stop lxd-bridge
systemctl restart lxd

And see if that fixes it?

@alci63

This comment has been minimized.

Copy link
Author

@alci63 alci63 commented Apr 6, 2016

I think I found the problem in the logs:

Apr  6 12:55:06 franck-ThinkPad-T430s kernel: [ 7029.101587] audit: type=1400 audit(1459940106.552:107): apparmor="DENIED" operation="mknod" profile="/usr/sbin/dnsmasq" name="/var/lib/lxd-bridge/dnsmasq.lxdbr0.leases" pid=22292 comm="dnsmasq" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Apr  6 12:55:06 franck-ThinkPad-T430s lxd-bridge.start[22255]: dnsmasq: ne peut ouvrir ou créer le fichiers de baux /var/lib/lxd-bridge//dnsmasq.lxdbr0.leases : Permission non accordée

So apparmor prevents dnsmasq to use lxd-bridge files... Indeed, I have the aa dnsmasq profile enforced. I don't know how packages are allowed to interact with apparmor profiles (ie could lxd packages modify aa profiles ?). Or if I should report a bug on apparmor-profiles. Or just modify it locally for myself and forget about it ?

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Apr 6, 2016

Oh, that would certainly explain it...

The apparmor dnsmasq profile should only apply to the system wide daemon (/etc/init.d/dnsmasq) and not to other daemons... so yeah that sounds like a bug for the dnsmasq package.

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Apr 6, 2016

or apparmor-profiles

@alci63

This comment has been minimized.

Copy link
Author

@alci63 alci63 commented Apr 6, 2016

I tested that switching from enforce to complain fixes the problem, and it does.
So I opened a bug against apparmor-profiles, as this seems to be the right way to do it according to https://wiki.ubuntu.com/DebuggingApparmor.
Here it is: https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1566944

I'm closing the case, as its apparmor related.

@alci63 alci63 closed this Apr 6, 2016
@rbelem

This comment has been minimized.

Copy link

@rbelem rbelem commented May 30, 2016

I had the same issue. The problem was that bind was listening on ipv6 :::53 and dnsmasq was trying to listen on fe80::aaaa:bbbb:cccc:53. That's why dnsmasq was getting permission denied.
My solution was to edit /etc/bind/named.conf.options and replace //listen-on-v6 { any; }; by listen-on-v6 { none; };

@Sparkxxx

This comment has been minimized.

Copy link

@Sparkxxx Sparkxxx commented Jun 4, 2016

Confirm rbelem solution - solved the problem that containers don't start anymore, 1 week ago I created the containers without any problem but today could not start them.

dpkg -l | grep lxd
ii lxd 2.0.2-0ubuntu116.04.1 amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.2-0ubuntu1
16.04.1 amd64 Container hypervisor based on LXC - client

sudo service lxd-bridge status
● lxd-bridge.service - LXD - network bridge
Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
Active: active (exited) since Sat 2016-06-04 12:30:52 EEST; 41min ago
Docs: man:lxd(1)
Process: 1045 ExecStart=/usr/lib/lxd/lxd-bridge.start (code=exited, status=0/SUCCESS)
Main PID: 1045 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CPU: 0
CGroup: /system.slice/lxd-bridge.service

Jun 04 12:30:52 nasosu systemd[1]: Starting LXD - network bridge...
Jun 04 12:30:52 nasosu lxd-bridge.start[1045]: dnsmasq: failed to create listening socket for 10.0.3.1: Address already in use
Jun 04 12:30:52 nasosu lxd-bridge.start[1045]: Failed to setup lxd-bridge.
Jun 04 12:30:52 nasosu systemd[1]: Started LXD - network bridge.

Ubuntu 16.04 LTS (GNU/Linux 4.4.0-22-generic x86_64)

@stgraber

This comment has been minimized.

Copy link
Member

@stgraber stgraber commented Jun 5, 2016

Yeah, it's a known problem and one that the bind maintainer in Ubuntu said he'd look into, by either not binding on all interfaces by default or doing some kind of soft-bind where the LXD dnsmasq would win when binding a specific IP.

@alioguzhan

This comment has been minimized.

Copy link

@alioguzhan alioguzhan commented Jun 18, 2016

Additionally for @rbelem 's answer;

if you don't have listen-on-v6 line in /etc/bind/named.conf.options file. you should add it as:

listen-on-v6 { none; };

And then restart lxd-brindge and networking:

service networking restart
service lxd-bridge restart

Then all should work fine.

@danielecr

This comment has been minimized.

Copy link

@danielecr danielecr commented Jun 30, 2016

Sorry, maybe is not the same kind of problem,
I am trying to bridge over a wlan connection managed by network-manager, so I can not define
auto wlan0
iface wlan0 inet manual

and
auto containerbr
iface containerbr inet dhcp
bridge_ports wlan0

does not create a containerbr0

Anyway, actually I would like to use lxdbr0 interface as setted with dpkg-reconfigure, but I have this:

service lxd-bridge status
● lxd-bridge.service - LXD - network bridge
   Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
   Active: active (exited) since gio 2016-06-30 12:40:14 CEST; 11min ago
     Docs: man:lxd(1)
  Process: 4162 ExecStart=/usr/lib/lxd/lxd-bridge.start (code=exited, status=0/SUCCESS)
 Main PID: 4162 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/lxd-bridge.service

giu 30 12:40:14 daniele-SATELLITE-L50-B systemd[1]: Starting LXD - network bridge...
giu 30 12:40:14 daniele-SATELLITE-L50-B lxd-bridge.start[4162]: dnsmasq: failed to create listening socket for 10.171.103.1: Indirizzo già in uso
giu 30 12:40:14 daniele-SATELLITE-L50-B lxd-bridge.start[4162]: Failed to setup lxd-bridge.
giu 30 12:40:14 daniele-SATELLITE-L50-B systemd[1]: Started LXD - network bridge.
giu 30 12:48:10 daniele-SATELLITE-L50-B systemd[1]: Started LXD - network bridge.

but nothing is using 10.171.103.1

ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 2c:60:0c:49:94:ef brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d0:7e:35:e8:43:f1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.139/24 brd 192.168.0.255 scope global dynamic wlan0
       valid_lft 85685sec preferred_lft 85685sec
    inet6 fe80::d27e:35ff:fee8:43f1/64 scope link 
       valid_lft forever preferred_lft forever
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 52:54:00:69:17:8c brd ff:ff:ff:ff:ff:ff
40: containerbr: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 76:3b:ee:74:28:62 brd ff:ff:ff:ff:ff:ff
@Sparkxxx

This comment has been minimized.

Copy link

@Sparkxxx Sparkxxx commented Jun 30, 2016

Bind9 is using port 53 on all interfaces and this is why the bridge doesn't start. Make bind listen only on the desired interfaces and the bridge will start. 

Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Daniele Cruciani notifications@github.com Date: 6/30/16 13:58 (GMT+02:00) To: lxc/lxd lxd@noreply.github.com Cc: Sparkxxx dbraileanu@gmail.com, Comment comment@noreply.github.com Subject: Re: [lxc/lxd] Cannot launch new container after switching to lxdbr0
(ubunut 16.04 upgrade) (#1857)
Sorry, maybe is not the same kind of problem,

I am trying to bridge over a wlan connection managed by network-manager, so I can not define

auto wlan0

iface wlan0 inet manual

and

auto containerbr

iface containerbr inet dhcp

bridge_ports wlan0

does not create a containerbr0

Anyway, actually I would like to use lxdbr0 interface as setted with dpkg-reconfigure, but I have this:

service lxd-bridge status
● lxd-bridge.service - LXD - network bridge
Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
Active: active (exited) since gio 2016-06-30 12:40:14 CEST; 11min ago
Docs: man:lxd(1)
Process: 4162 ExecStart=/usr/lib/lxd/lxd-bridge.start (code=exited, status=0/SUCCESS)
Main PID: 4162 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CPU: 0
CGroup: /system.slice/lxd-bridge.service

giu 30 12:40:14 daniele-SATELLITE-L50-B systemd[1]: Starting LXD - network bridge...
giu 30 12:40:14 daniele-SATELLITE-L50-B lxd-bridge.start[4162]: dnsmasq: failed to create listening socket for 10.171.103.1: Indirizzo già in uso
giu 30 12:40:14 daniele-SATELLITE-L50-B lxd-bridge.start[4162]: Failed to setup lxd-bridge.
giu 30 12:40:14 daniele-SATELLITE-L50-B systemd[1]: Started LXD - network bridge.
giu 30 12:48:10 daniele-SATELLITE-L50-B systemd[1]: Started LXD - network bridge.

but nothing is using 10.171.103.1

ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 2c:60:0c:49:94:ef brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d0:7e:35:e8:43:f1 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.139/24 brd 192.168.0.255 scope global dynamic wlan0
valid_lft 85685sec preferred_lft 85685sec
inet6 fe80::d27e:35ff:fee8:43f1/64 scope link
valid_lft forever preferred_lft forever
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 52:54:00:69:17:8c brd ff:ff:ff:ff:ff:ff
40: containerbr: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 76:3b:ee:74:28:62 brd ff:ff:ff:ff:ff:ff


You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

@danielecr

This comment has been minimized.

Copy link

@danielecr danielecr commented Jul 1, 2016

@Sparkxxx thank you for reply, actually yesterday I solved this by searching for dnsmasq, discovering it is a dns server, and uninstalling bind server (for now, maybe I will re-enable it listening only on 127.0.0.1)

Unfortunately no introductory blog post mentioned dnsmasq and the fact that lxd solve the name automatically setting a resolver, I will comment on as much post I found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
7 participants
You can’t perform that action at this time.