Let's Encrypt renewal fails with integrated Caddy server

[startaq]

We are running the snap version of Rocket.Chat and are using Caddy as reverse proxy. Today it wanted to renew the certificates, but this failed with the error:

kernel: audit: type=1400 audit(1495005754.301:286): apparmor="DENIED" operation="mkdir" profile="snap.rocketchat-server.rocketchat-caddy" name="/root/snap/rocketchat-server/653/.caddy/" pid=25685 comm="caddy" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

This caused the startup of Caddy to fail and thus our Rocket.Chat instance was unavailable. I've switched to self-signed certificates for now, this allows Caddy to start.

Base system is Ubuntu 16.04.2 LTS.

Rocket.Chat Version: 0.56
Running Instances: 1
DB Replicaset OpLog: Enabled
Node Version: v4.8.1

[startaq]

Found the issue. For some reason the AppArmor profiles in /var/lib/snapd/apparmor/profiles/ were not updated. Manually updating the SNAP_REVISION variable in the profiles to refer to the current version solved it.

[geekgonecrazy]

@startaq can you do: snap version or snap --version and get me the version number? If this is a newer version of snapd we might want to get this reported upstream

[zyga]

Hello, snapd developer here. Can you tell me more about the version of snapd and the operations you performed? Ideally you may still have snap changes that describes the relevant update operation.

[startaq]

snap version was 2.24, it was upgraded to 2.25 after the issues started. Currently installed is:

snap    2.25
snapd   2.25
series  16
ubuntu  16.04
kernel  4.8.14-apparmor-2

What I think happened was: A new version of the Rocket.Chat snap was installed (automatically), but the AppArmor profiles in /var/lib/snapd/apparmor/profiles/ were not updated accordingly.

All I did to make it work again was changing the variable SNAP_REVISION at the top of the following files:

snap.rocketchat-server.backupdb
snap.rocketchat-server.initcaddy
snap.rocketchat-server.rocketchat-caddy
snap.rocketchat-server.rocketchat-mongo
snap.rocketchat-server.rocketchat-server

To the currently installed revision (653). The previously installed revision was 616. Then reloading the profile with apparmor_parser -r <profile> and restarting the service with systemctl.

[zyga]

I see what the problem is.

Snapd probed your kernel and decided that it doesn't support apparmor sufficiently and decided not to generate such profiles. The rest of the system still imported the previous profiles into the kernel.

Your kernel doesn't look like stock Ubuntu kernel and probably doesn't have all the required patches. To fix the issue either remove all the apparmor profiles that are around the system (or boot with apparmor disabled) or (better option) use the stock Ubuntu kernel.

[startaq]

This is a Scaleway Ubuntu VM, using their 4.8.14-apparmor-2 bootscript (which supplies the kernel). There currently is no way to use a standard Ubuntu kernel. The kernel config can be found here: https://github.com/scaleway/kernel-tools/blob/master/x86_64/4.8.14-apparmor/.config

Why does snapd probe the kernel? It would probably be better to check if AppArmor is installed, and if yes, generate the profiles. This would avoid such an issue. Or at least generate a warning.

So, until there is a way to use the official Ubuntu kernel, it's probably best to try to run Snap without AppArmor?

[zyga]

Snapd probes the kernel because we need specific apparmor features. We look if all of them are available before choosing to enable apparmor. The problem is that ubuntu kernel carries additional patches that snapd relies on. They are being upstreamed (the set shrinks with each release) but it is not all there yet. Many security features rely on apparmor being able to do new things.

[zyga]

As for disabling apparmor, given that Scaleway claim to offer Ubuntu and uses the official Ubuntu repository there is no way to disable apparmor. The reality is that this is not the real Ubuntu without the real Ubuntu kernel.