Mysql, Privileged mode, cannot open shared object file

[dalguete]

Well, this is like a duplicate for the issue #5430, but as that is marked as closed, I'm creating this new one to catch your attention.

The problem seems to be the privileged mode. When I run the container without that mode, it runs fine, but with that mode, it throws this:

/usr/sbin/mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: Permission denied

I'm using a Ubuntu 14.04, 64 bits in host and container's image.

And privileged mode here, is necessary (per dev env requirements).

[thaJeztah]

To speed up resolving your issue, it might help to add the output of docker version and docker info, as well as the content of the Dockerfile of the container you're running (if possible)

[dalguete]

Sure, Docker version is

Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

Docker info:

Containers: 4
Images: 115
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 123
Execution Driver: native-0.2
Kernel Version: 3.13.0-32-generic
Username: dalguete
Registry: [https://index.docker.io/v1/]

And details about how to reproduce this, in the next gist https://gist.github.com/dalguete/cd2dc5471530b8df00da

[thaJeztah]

Thanks, I'm not able to help you out personally here, but this might give some insight to the people who do :)

[grahamc]

Hypothesizing - is mysql trying to load the aio kernel module? If this is the case, you would need to load the module before starting the container.

[dalguete]

But why that happens under privileged mode and not under non-privileged mode? The other way around seems to be a more plausible place to have these errors.

If the case you mention is true, mysql (and maybe other services/programs) are doing more stuff under less controlled envs, but in the end, that has to be contained. Still looks like a bug to me.

[grahamc]

Oops, I rescind my comment. I had the situation reversed.

[crosbymichael]

This maybe an apparmor issue if it only happens in privileged mode. Do you have the mysql apparmor profiles installed on your host machine?

[dalguete]

I think I do because I have mysql installed in host. But why is that? I mean, why apparmor could be involved in the problem here under one mode and not under the other? It seems you've faced problems like this before; can you elaborate a bit?

[crosbymichael]

@dalguete

This is because apparmor applies profiles based on the binary paths. When we run the container in privileged mode docker only tells apparmor that we are not setting the profile so leave this unconfined. However, by not specifying a profile, apparmor looks at the binary path and sees if it has any profiles matching the binary and automatically applies them.

A few things that I would suggest you doing is not have the profiles installed on your host when using apparmor if you are running everything in containers.

The other is you should not run a database container in privileged mode. Mysql should not need extra capabilities and you don't want to open up access to your host for a database that does not require it. Very few applications actually require privileged mode and mysql is definitely not one of them.

[dalguete]

That is quite interesting indeed. To be honest I didn't even know that privileged touches AppArmor. After some reading I found this https://docs.docker.com/reference/run/#runtime-privilege-and-lxc-configuration

I'm gonna do some tests, to verify some theories I have, but if I understand it well, when you run under privileged mode, AppArmor profiles are not applied inside the container, but as I have installed mysql in host, and because of the privilged mode, the AppArmor profiles in host are applied hoping to see calls from host, not from container and that's when the errors occur. Is correct my reasoning here?

This will be part of my tests, but if the case above is true, uninstalling mysql in host will do the magic I think, and mysql would be able to run under a privileged mode (hope so).

The privileged mode is required per a experimental dev env I'm building, meant to be used in local machines only. Mostly because of services installed there (web server) and its use of data in volumes (to make that available in host for editing, and inside container, to see it work). Here is some info about the reasons for that, https://github.com/dalguete/docker/tree/master/bindfs.

Maybe a warning message or something could be necessary when running a container under privileged mode, that reminds us what to be aware of in that mode (maybe a link to a web page or something), because it wasn't pretty obvious at first.

[crosbymichael]

Yes, you should uninstall mysql from your host and make sure that it also removes the profile.

I think with docker 1.2 you will be able to do what you need without --privileged We have two new flags --device and --cap-add that will allow you to add the ability to work with extra devices and add caps so you can mount and such.

Just remember that the more privileges you give to a container the more holes you poke in it causing it to be less secure. There is no zero sum tradeoff.

[crosbymichael]

I'm closing this because it's an apparmor issue with your current setup and there is already an issue for this exact problem.

[dalguete]

Posting this in case anyone later faces something similar. I did the testing as stated before:

"This will be part of my tests, but if the case above is true, uninstalling mysql in host will do the magic I think, and mysql would be able to run under a privileged mode (hope so)."

Test PASSED. Hopefully by the time someone hits the same issues, --device and --cap-add will be already available.

[juanman2]

Thanks this was useful, we ran into the same thing in a very different configuration.

[mikz]

it is "dangerous" but for anyone who would like to solve it for dev machines:

sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

[econtal]

It's been a while, but for people who ends up here and don't want to disable AppArmor on their host, it's probably better to load AppArmor profile unconfined instead or any custom policy¹:

docker run ... --security-opt apparmor=unconfined yourimage

See more details here: #38420

In particular, this is how kubernetes-in-docker is doing by default: https://github.com/kubernetes-sigs/kind/pull/1183/files

¹ in certain versions the custom policy docker-unconfined was created for this usage. But this has been removed recently.

[NoelCarlton]

it is "dangerous" but for anyone who would like to solve it for dev machines:

sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld

Interesting, and what should I do with Centos7 which hosts docker? I'm encountering this same issue.