lxd (snap) failed to start after snap refresh

[olivierfraysse]

This morning after the snap refresh (2017-05-01T19:33:24Z) LXD crashes.
I (sorry) took the opportunity of this downtime to upgrade kernel and reboot.

Unfortunately after this reboot, LXD failed to start :

# lxd --debug --group lxd
cannot perform operation: mount /var/lib/snapd/hostfs/var/lib/lxd /var/lib/lxd -o nosuid,nodev,noexec,rbind,rslave: Permission denied

• snap-confine 2.25 (I I suspect this but I do not know why)
• snappy core 16-2 rev 1689
• snappy lxd 2.12 rev 2071 (but the problem is the same after snap revert lxd)
• storage zfs (https://pastebin.com/2p4cW7ee )

# snap version
snap    2.25
snapd   2.25
series  16
ubuntu  16.04
kernel  4.4.0-77-generic

[brauner]

@stgraber you might be best-suited to know what's going on here.

[olivierfraysse]

# cat /var/log/apt/history.log

Start-Date: 2017-05-02 11:37:29
Commandline: apt dist-upgrade
Requested-By: olive (1000)
Install: linux-image-4.4.0-77-generic:amd64 (4.4.0-77.98, automatic), linux-image-extra-4.4.0-77-generic:amd64 (4.4.0-77.98, automatic), linux-headers-4.4.0-77-generic:amd64 (4.4.0-77.98, automatic), linux-headers-4.4.0-77:amd64 (4.4.0-77.98, automatic)
Upgrade: libdns-export162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), libisccfg140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), linux-headers-generic:amd64 (4.4.0.73.79, 4.4.0.77.83), linux-libc-dev:amd64 (4.4.0-73.94, 4.4.0-77.98), linux-image-generic:amd64 (4.4.0.73.79, 4.4.0.77.83), snapd:amd64 (2.23.6, 2.25), libirs141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), bind9-host:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), snap-confine:amd64 (2.23.6, 2.25), dnsutils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), libisc160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), bind9utils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), dpkg:amd64 (1.18.4ubuntu1.1, 1.18.4ubuntu1.2), libisc-export160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), liblwres141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), distro-info-data:amd64 (0.28ubuntu0.2, 0.28ubuntu0.3), bind9:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), libdns162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), logrotate:amd64 (3.8.7-2ubuntu2, 3.8.7-2ubuntu2.16.04.1), libfreetype6:amd64 (2.6.1-0.1ubuntu2.1, 2.6.1-0.1ubuntu2.2), libisccc140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), libbind9-140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.5, 1:9.10.3.dfsg.P4-8ubuntu1.6), libdpkg-perl:amd64 (1.18.4ubuntu1.1, 1.18.4ubuntu1.2), linux-generic:amd64 (4.4.0.73.79, 4.4.0.77.83), dpkg-dev:amd64 (1.18.4ubuntu1.1, 1.18.4ubuntu1.2)
End-Date: 2017-05-02 11:38:05

[lool]

In Olivier's report, he indicates 2.12 snap rev 2071 and that a revert doesn't help; note that these versions are his versions after the revert. His problems seemingly started after the automatic refresh to 2.13 (rev 2071 of the snap)

He also confirmed to me that the lxd interface was properly connected:

snap interfaces |grep lxd

:home lxd
:lxd-support lxd
:network lxd
lxd:lxd

[stgraber]

@olivierfraysse can you post the output of "dmesg"?

[stgraber]

Just confirmed that a clean install of the snap works, so it doesn't appear to be a global regression at least.

[olivierfraysse]

# dmesg
http://paste.ubuntu.com/24498352/

[olivierfraysse]

# for stuff in /etc/apparmor.d/*snap*;do echo $stuff;cat $stuff | pastebinit;done
/etc/apparmor.d/snap.core.1689.usr.lib.snapd.snap-confine
http://paste.ubuntu.com/24498364/
/etc/apparmor.d/usr.lib.snapd.snap-confine.dpkg-bak
http://paste.ubuntu.com/24498365/
/etc/apparmor.d/usr.lib.snapd.snap-confine.real
http://paste.ubuntu.com/24498366/

[stgraber]

This is a snapd issue which we've been discussing on IRC with the snapd guys.
I'm closing this issue as there's no LXD action for this but we'll keep in touch on IRC and try to figure out what happened here.

[olivierfraysse]

the solution was... to delete this symlink : /var/lib/lxd ( -> /var/snap/lxd/common/lxd/)
no idea who create this symlink.