cap_set_file not permitted on aufs storage driver only

[jakedt]

When running this minimal Dockerfile:

FROM centos
RUN yum install -y httpd mod_ssl

The following error occurs only when using the aufs storage driver:

Installing : httpd-2.4.6-17.el7.centos.1.x86_64                           8/9
Error unpacking rpm package httpd-2.4.6-17.el7.centos.1.x86_64

error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file

When running with the devicemapper driver the command succeeds successfully. The capabilities listed for the process are identical under both storage drivers and include CAP_SETFCAP and CAP_FOWNER which are listed by set_cap_file as requirements.

Here is the full list of capabilities which both processes have:

Capabilities for `11004': =ep cap_chown,cap_dac_override,cap_fowner,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_setfcap+i cap_dac_read_search,cap_fsetid,cap_linux_immutable,cap_net_broadcast,cap_net_admin,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_lease,cap_audit_write,cap_audit_control,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend-ep

[tiborvass]

@jakedt Can you please paste the output of docker version and docker info ?

[jakedt]

Sure thing!

[sudo] password for jake: 
Client version: 1.1.1
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): bd609d2
Server version: 1.1.1
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): bd609d2

Containers: 5
Images: 356
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 367
Execution Driver: native-0.2
Kernel Version: 3.13.0-24-generic
Debug mode (server): true
Debug mode (client): false
Fds: 10
Goroutines: 9
EventsListeners: 0
Init Path: /usr/bin/docker
Sockets: [unix:///var/run/docker.sock]
WARNING: No swap limit support

Of course the storage driver is just the one I happen to use by default.

[tiborvass]

Ping @crosbymichael @vieux

[henning]

This topic is already blogged (and from the blog post also communicated to the docker dev team): http://major.io/2014/03/26/docker-trusted-builds-and-fedora-20/ meanwhile, I'm too still searching for a way to install httpd on a centos based docker image... iputils (installed when you try to install the development package group) has the same issue

[shykes]

@henning feel free to contribute to the upstream docs, so that the wider Docker community can benefit.

[henning]

@shykes: not clear what you mean, but sure, as soon as i have a solution or
a workaround i do so.
For now i'm still trying to understand whats going on.

[henning]

Some more research:

• running docker with --privileged isn't helping, plus, --privileged option is not available for the build command (maybe because it's runing privileged anyway?)
• there's another bug with the same symptom said to be fixed already: We seem to have lost CAP_SET_FILE #5928
• Theres also an info on cap_set_file stuff in Linux - it might mean that the host OS Kernel needs to be specifically compiled, so at least in my case when running centos7 in a container on an Ubuntu hist might not work? http://rglinuxtech.com/?p=140
• Redhat Bugs related to cap_set_file:
  • https://bugzilla.redhat.com/show_bug.cgi?id=1012952
  • https://bugzilla.redhat.com/show_bug.cgi?id=648654 (only relevant if docker uses tmpfs somewhere inside)

Next thing I'll try is checking if there's any different behaviour when running docker on an centos7 host.

[jamshid]

Just encountered this, it is breaking my "FROM centos" Dockerfiles after upgrading to current boot2docker.

$ docker run -ti centos bash
bash-4.2# yum install -y java-1.7.0-openjdk
...
  Installing : iputils-20121221-6.el7.x86_64                              45/80 
Error unpacking rpm package iputils-20121221-6.el7.x86_64
error: unpacking of archive failed on file /usr/bin/ping: cpio: cap_set_file
  Installing : fontpackages-filesystem-1.44-8.el7.noarch                  46/80 
error: iputils-20121221-6.el7.x86_64: install failed
...
Failed:
  iputils.x86_64 0:20121221-6.el7                                               
Complete!

Not sure what the best fix is (using --privileged didn't help), so for now I changed my "FROM centos" to "FROM centos:centos6".

[jaenswart]

same for me,

xorg-x11-fonts-Type1.noarch 0:7.5-9.el7

Failed:
iputils.x86_64 0:20121221-6.el7

Complete!
2014/07/25 16:02:54 The command [/bin/sh -c yum install -y java-1.7.0-openjdk which telnet unzip openssh-server sudo openssh-clients] returned a non-zero code: 1

[crosbymichael]

From looking into this it appears to be an AUFS specific bug which means fixing it requires patching AUFS and recompiling the kernel. I suggest using BTRFS or devicemapper instead because we really have nothing to change in docker to fix this issue.

@unclejack thoughts?

[jgangemi]

i'm having the same issue as @jamshid abd @jaenswart - coud this is a bug in centos7 itself caused by a recent update. a co-worker didn't have this problem against a centos7 image from ~5-6 (maybe a little more/less) weeks ago but he now sees this after a 'docker pull centos'.

centos6 doesn't seem to exhibit this issue.

[bruceadams]

I see no way around this problem when my host system is Ubuntu 14.04 (both physical and a VM). However it works fine in a CentOS 7 VM host system. So, doing the docker build on a CentOS 7 host is a workaround for this issue. (I just used a small DigitalOcean droplet for my build.)

I see the problem when doing a "yum update" in a CentOS 7 docker image, such as:

$ docker run -i -t --rm centos:centos7 /bin/bash
bash-4.2# yum update -y

I only see errors on Ubuntu host systems. The error messages (amidst a bunch of output from yum) are:

Error unpacking rpm package systemd-208-11.el7_0.2.x86_64
error: unpacking of archive failed on file /usr/bin/systemd-detect-virt: cpio: cap_set_file

Once I've created a CentOS 7 image on a CentOS 7 host system, I can use the image just fine on Ubuntu, so long as I don't need to do a further yum install of any problematic package.

[omidraha]

I have same problem, my host system is Linux debian 3.16-2-amd64 and my image is CentOS 7.

bash-4.2# yum install -y iputils

Running transaction
  Installing : iputils-20121221-6.el7.x86_64
Error unpacking rpm package iputils-20121221-6.el7.x86_64
error: unpacking of archive failed on file /usr/bin/ping: cpio: cap_set_file
  Verifying  : iputils-20121221-6.el7.x86_64

Failed:
  iputils.x86_64 0:20121221-6.el7 

bash-4.2# yum install -y systemd

Running transaction
  Updating   : systemd-208-11.el7_0.2.x86_64
Error unpacking rpm package systemd-208-11.el7_0.2.x86_64
error: unpacking of archive failed on file /usr/bin/systemd-detect-virt: cpio: cap_set_file
systemd-208-11.el7.x86_64 was supposed to be removed but is not!
  Verifying  : systemd-208-11.el7.x86_64
  Verifying  : systemd-208-11.el7_0.2.x86_64

Failed:
  systemd.x86_64 0:208-11.el7  systemd.x86_64 0:208-11.el7_0.2

[metal3d]

@crosbymichael is right. AUFS is the problem. I had a "solution" (workaround) on debian and maybe ubuntu...

Export your images and/or containers (because they will be unaccessibles unless you revert the following instructions)

service docker stop 

Open "/etc/default/docker" and set DOCKER_OPTS with

DOCKER_OPTS="-s devicemapper"

Then relaunch docker service

service docker start

Now, Docker is using devicemapper and not AUFS, that should stop to complain with yum and httpd (or other problem with cap_set_file).

I was able to retrieve my images and containers by removing the modification in /etc/default/docker and restarting docker service. I guess that this modification can be dagerous, so please make images and containers backups before to test.

Now... who can manage this bug ? docker team ? AUFS team ?

[asinus666]

I do see the same issue when trying to install httpd in a centos container though my docker runs using devicemapper as storage backend:

[root@30b87986efbd ~]# yum -y install httpd
[...]
Running transaction
Installing : httpd-2.4.6-18.el7.centos.x86_64 1/1
Error unpacking rpm package httpd-2.4.6-18.el7.centos.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file
Verifying : httpd-2.4.6-18.el7.centos.x86_64 1/1
[...]

breuerma@data ~ $ docker info
Containers: 39
Images: 305
Storage Driver: devicemapper
Pool Name: docker-8:1-1179116-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 12.8 GB
Data Space Total: 107.4 GB
Metadata Space Used: 16.54 MB
Metadata Space Total: 2.147 GB
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Kernel Version: 3.17.3-data-8.1
Operating System:

breuerma@data ~ $ docker version
Client version: 1.3.1
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 4e9bbfa
OS/Arch (client): linux/amd64
Server version: 1.3.1
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 4e9bbfa

OS is gentoo.

[fogfish]

I used to have same issue but with librsvg2-tools. The problem has been solved by removing all centos and re-create app images from scratch.

[metal3d]

The other solution I will use from now: not use Debian as host. My other server that is a CentOS server has no problem to use debian, centos or other images/containers + installation.

[tianon]

I use Debian + BTRFS, and it works swell. ;)

[metal3d]

I feel the troll coming behind me ;)
BTRFS ok, but RAID support is not fully implemented and I don't know it enought to install it on my production servers without any risk.
At this time, I'm using LVM/ext4 with CentOS and everything is ok. That's not against Debian, but I have less problem and more solutions and knowledge on CentOS (and Fedora). Maybe one day I'll come back ;)

BTW: yes, I will deeply try BTRFS in next weeks

[SvenDowideit]

@metal3d you can also use devicemapper on debian - that way its the same as your centos default.

[metal3d]

@SvenDowideit This is exactly what I said 2 comments earlier ;) #issuecomment-62723470

Le Thu Nov 27 2014 at 07:21:42, Sven Dowideit notifications@github.com a
écrit :

@metal3d https://github.com/metal3d you can also use devicemapper on
debian - that way its the same as your centos default.

—
Reply to this email directly or view it on GitHub
#6980 (comment).

[SvenDowideit]

@metal3d ah - that makes your comment in #6980 (comment) that your solution was not to use debian even more confusing.

[metal3d]

Yes, my other comment was for @fogfish that removes the whole centos containers. That was for trolling ;) I consider that removing containers that use CentOS because Debian cannot use them is a worse solution than removing Debian host.
Yes, I don't like Debian :p (troll inside)

[fogfish]

No trolling here :)

I have not meet an issue with cap_set_file at production (container runs on top of AWS Linux, which is derived from CentOS) but cap_set_file have ruined my development environment. I am using Boot2Docker on MacOS, it uses CentOS as host. I've tried to build a container with librsvg2-tools. The container was derived from centos:latest. The build procedure failed with same reason as described above.

The quick fix was to clean up cache and remove all centos:latest images and all derived image from my dev env cache, after that problem with cap_set_file magically disapperars once latest centos was downloaded).

BTW, dev uses aufs but prod is dev-mapper. I guess dev-mapper is the way to address this issue like it was suggested.

[metal3d]

@fogfish that was me who was trolling :) I was joking.
What I meant is that it's a pitty to drop CentOS contianers because of a Debian problem.
I understand your point of view and you are absolutly right.
Anyway, the problem should not appear using devicemapper or BTRFS, I will try to move my EXT4 partition to BTRFS tomorow.
EDIT: I just understand that your host is CentOS... so it's not the same problem as mine
EDIT2: And now I re-read your comments, I understand that you didn't dropped CentOS: I really have to read twice - I'm not english

[hwongcom]

Ok, I see the problem here as well. What's the solution again for CentOS 7? Are you guys saying switch to the slower dev-mapper? THAT'S a solution?!?!

Come on, how to solve this without switching to dev-mapper?

I need CentOS 7. Solutions please! Thanks!

[jessfraz]

overlayfs is being merged in Docker 1.4 and the 3.18 kernel, this would
also give you the opportunity to upgrade your kernel to something other
than a 2.6.x kernel.....

On Wed, Dec 3, 2014 at 2:06 AM, hwongcom notifications@github.com wrote:

Ok, I see the problem here as well. What's the solution again for CentOS
7? Are you guys saying switch to the slower dev-mapper? THAT'S a
solution?!?!

Come on, how to solve this without switching to dev-mapper?

—
Reply to this email directly or view it on GitHub
#6980 (comment).

[The-Judge]

This happens on Docker's own infrastructure, when using Automated Builds; see here for example. Changing the local Build Host or storage driver seems like a poor workaround, since it only enables one to build the image locally.

[thaJeztah]

@The-Judge have you contacted support@docker.com for that? Perhaps the support team for docker hub can look into that

[matanster]

Same problem here so it seems (client version 1.10.1). Trying to get a grip of what's going on ―

What does it mean:

The following error occurs only when using the aufs storage driver

Doesn't docker use aufs by default?

Now, Docker is using devicemapper and not AUFS

― as per the workaround suggested above, what are the overall ramifications of switching docker to devicemapper?

[thaJeztah]

@matanster see the storage driver section of the docs for more in-depth information on the differences between drivers https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

[NagoLazaro]

+1 , centos:7 container on a ubuntu 15.10 host

[martinsbalodis]

This problem isn't fixed in ubuntu 16.04.

[ttroy50]

I've also seen the problem building a centos:7 image on an ubuntu 16.04 host.

To use the devicemapper workaround on 16.04 you can do the following

systemctl stop docker

Edit /lib/systemd/system/docker.service to add -s devicemapper to the ExecStart command. See https://docs.docker.com/engine/admin/systemd/

reload the service file

systemctl daemon-reload

start docker

systemctl start docker

[crashbit666]

docker/hub-feedback#461 In this thread solves the problem

[josdotso]

This issue went away for me when I upgraded to Ubuntu 16.04.1 and rebooted.

[espoelstra]

Has anybody here tested the workaround from docker/hub-feedback#461 (comment) ?

Experienced same issue, cannot install httpd in a centos container

error unpacking rpm package httpd-2.4.6-45.el7.centos.4.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec;597bcbad: cpio: cap_set_file
error: httpd-2.4.6-45.el7.centos.4.x86_64: install failed

With the image centos/https, they set --setopt when installing the package:

RUN yum -y --setopt=tsflags=nodocs update &&
yum -y --setopt=tsflags=nodocs install httpd &&
yum clean all

Done so in my Dockerfile, now it works fine.