Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AUFS broken permissions? #20240

Closed
dmp42 opened this issue Feb 11, 2016 · 20 comments
Closed

AUFS broken permissions? #20240

dmp42 opened this issue Feb 11, 2016 · 20 comments
Labels
area/kernel area/storage/aufs kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.

Comments

@dmp42
Copy link
Contributor

dmp42 commented Feb 11, 2016

docker info

Containers: 7
 Running: 0
 Paused: 0
 Stopped: 7
Images: 59
Server Version: 1.10.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 51
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.17-boot2docker
Operating System: Boot2Docker 1.10.0 (TCL 6.4.1); master : b09ed60 - Thu Feb  4 20:16:08 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.2 MiB
Name: pouet2
ID: 6J3W:3S7J:QA6U:2JLY:PEPW:6WNL:4P6N:PWFB:U2XT:KP3S:HDFT:E6QU
Debug mode (server): true
 File Descriptors: 10
 Goroutines: 22
 System Time: 2016-02-11T18:32:54.290248331Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Username: dmp42
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox

Dockerfile:

FROM debian:jessie

# Creating test account
RUN adduser --quiet testing

# Copy contrib folder
COPY ./ /home/testing/
RUN chown -R testing:testing /home/testing/

RUN ls -lA /home/testing/contrib

USER testing

RUN ls -lA /home/testing/contrib

To reproduce, be sure to have a contrib folder in the current directory, with at least one subfolder.

Result:

docker build .

Step 5 : RUN ls -lA /home/testing/contrib
 ---> Running in 14b2711f2a50
total 20
drwx------ 2 testing testing 4096 Feb 11 18:31 apache
drwxr-xr-x 2 testing testing 4096 Feb 11 18:31 ceph
drwx------ 4 testing testing 4096 Feb 11 18:31 compose
drwxr-xr-x 4 testing testing 4096 Feb 11 18:31 docker-integration
drwx------ 6 testing testing 4096 Feb 11 18:31 testing
 ---> 4b3a7bdb66d0
Removing intermediate container 14b2711f2a50
Step 6 : USER testing
 ---> Running in c3b0cb5a323e
 ---> c2742d90a5fb
Removing intermediate container c3b0cb5a323e
Step 7 : RUN ls -lA /home/testing/contrib
 ---> Running in 789914bc918e
ls: cannot access /home/testing/contrib/testing: Permission denied
ls: cannot access /home/testing/contrib/compose: Permission denied
ls: cannot access /home/testing/contrib/docker-integration: Permission denied
ls: cannot access /home/testing/contrib/apache: Permission denied
ls: cannot access /home/testing/contrib/ceph: Permission denied
total 0
d????????? ? ? ? ?            ? apache
d????????? ? ? ? ?            ? ceph
d????????? ? ? ? ?            ? compose
d????????? ? ? ? ?            ? docker-integration
d????????? ? ? ? ?            ? testing
The command '/bin/sh -c ls -lA /home/testing/contrib' returned a non-zero code: 1
@dmp42
Copy link
Contributor Author

dmp42 commented Feb 11, 2016

Ohhhh shut-up gordon, docker version is in there :)
And yes, it's virtualbox.

@tonistiigi
Copy link
Member

fwiw I couldn't reproduce(no boot2docker): https://gist.github.com/tonistiigi/d212392807dfd1e626ee

@AkihiroSuda
Copy link
Member

I can also reproduce this (B2D 1.10, Ubuntu 15.10(kernel 4.2.0-27-generic)), and verified that

  • For reproduction, the permission of contrib needs to be 700
  • COPY is not needed for reproduction.

Dockerfile:

FROM debian:jessie
RUN adduser --quiet testing
RUN mkdir -p /home/testing/d1/d2
RUN chmod 700 /home/testing/d1
RUN ls -lAR /home/testing
RUN chown -R testing:testing /home/testing/
RUN ls -lAR /home/testing

USER testing
RUN ls -lAR /home/testing

@tonistiigi
Copy link
Member

This is really weird. I can make file accessible to the user by visiting them with root.

» docker run -it 10fb28eafe24770f2 bash
root@952e01d47881:/# ls -lR /home/testing/
/home/testing/:
total 4
drwx------ 5 testing testing 4096 Feb 12 06:52 d1

/home/testing/d1:
total 4
drwx------ 2 testing testing 4096 Feb 12 06:52 d2

/home/testing/d1/d2:
total 0
root@952e01d47881:/# su testing
testing@952e01d47881:/$ ls -lR /home/testing
/home/testing:
total 4
drwx------ 5 testing testing 4096 Feb 12 06:52 d1

/home/testing/d1:
total 4
drwx------ 2 testing testing 4096 Feb 12 06:52 d2

/home/testing/d1/d2:
total 0
testing@952e01d47881:/$ exit
root@952e01d47881:/# exit

» docker run -it 10fb28eafe24770f2 bash
root@3258ca8167c6:/# su testing
testing@3258ca8167c6:/$ ls -lR /home/testing/
/home/testing/:
total 4
drwx------ 5 testing testing 4096 Feb 12 06:52 d1

/home/testing/d1:
ls: cannot access /home/testing/d1/d2: Permission denied
total 0
d????????? ? ? ? ?            ? d2
ls: cannot open directory /home/testing/d1/d2: Permission denied

@thaJeztah thaJeztah added area/storage/aufs kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. labels Feb 12, 2016
@thaJeztah
Copy link
Member

I think I've seen a similar issue a while back (also same behavior, i.e., visiting as root magically made the permissions snap), searching if I can find it

@thaJeztah
Copy link
Member

Think it was this one; #13823 (comment), which was related to incorrect umask being set, and resolved by #13941

@otherjason
Copy link

I can confirm I've had the same issue that was presented here in my application. I'm using Ubuntu 14.04 with the v3.19 kernel update package installed (and AUFS, obviously).

Like @thaJeztah and @tonistiigi pointed out, if you visit the directory as root, then the permissions start working. I haven't figured out a way to use that attribute to reliably work around the problem, though. I tried inserting commands in my Dockerfile to visit the directories as root, but the magic effect goes away when I create child containers from the resulting image.

@AkihiroSuda
Copy link
Member

I can still reproduce the bug with Boot2Docker 1.10.2 + AUFS 20160223 (sfjro/aufs4-standalone@164e70a)

@yajo
Copy link

yajo commented Feb 25, 2016

Workaround: Run ls -R /home/testing as root before any other command.

@yajo
Copy link

yajo commented Feb 25, 2016

This is weird.

My Linux host machine runs this:

Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-6.git6ec29ef.fc23.x86_64
 Go version:      go1.5.3
 Git commit:      7206621
 Built:           Mon Jan 25 20:52:22 UTC 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-6.git6ec29ef.fc23.x86_64
 Go version:      go1.5.3
 Git commit:      7206621
 Built:           Mon Jan 25 20:52:22 UTC 2016
 OS/Arch:         linux/amd64

And the failing boot2docker was running this:

Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-6.git6ec29ef.fc23.x86_64
 Go version:      go1.5.3
 Git commit:      7206621
 Built:           Mon Jan 25 20:52:22 UTC 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.2
 API version:     1.22
 Package version: 
 Go version:      go1.5.3
 Git commit:      c3959b1
 Built:           2016-02-22T22:37:33.778002647+00:00
 OS/Arch:         linux/amd64

That's why I decided to create a docker-machine with:

docker-machine create --driver virtualbox --virtualbox-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v1.9.1/boot2docker.iso test0-1.9.1

Whose version is:

Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-6.git6ec29ef.fc23.x86_64
 Go version:      go1.5.3
 Git commit:      7206621
 Built:           Mon Jan 25 20:52:22 UTC 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.9.1
 API version:     1.21
 Package version: 
 Go version:      go1.4.3
 Git commit:      a34a1d5
 Built:           Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:         linux/amd64

... And the bug still happens!

@bmbroom
Copy link

bmbroom commented Apr 5, 2016

I came across a different bug, but it's similar enough I might have chosen the same title. I have a docker image in which a non-root user cannot create any file in a specific directory to which that user supposedly has permission. Accessing the directory as root does not fix the problem, nor does changing the directory's attributes. However, renaming it (even as the non-root user) fixes it. Renaming it back to the original name also works. This is the only directory for which I've noticed the issue.

I noticed this bug on a Google compute engine node to which I added docker (with AUFS):

Docker info:

Containers: 11
 Running: 4
 Paused: 0
 Stopped: 7
Images: 25
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 182
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.19.0-56-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.465 GiB
Name: instance-2
ID: YPAN:7Z53:SIQG:7HCF:PASJ:A4AP:O4YO:KOUL:OUTS:3G4G:DFGC:VYU3

Dockerfile:

FROM bmbroom/ngchmmanagerdefault:permissionsbug
RUN ls -al /opt/glassfish3/glassfish/domains/domain1/config
RUN id
RUN touch /opt/glassfish3/glassfish/domains/domain1/config/xxxx

Output of docker build:

Sending build context to Docker daemon 4.096 kB
Step 1 : FROM bmbroom/ngchmmanagerdefault:permissionsbug
 ---> 1da8c2ae5b07
Step 2 : RUN ls -al /opt/glassfish3/glassfish/domains/domain1/config
 ---> Running in 8d6962d314df
total 272
drwx------  2 chmuser     999  4096 Oct  8 13:43 .
drwxr-xr-x 21 chmuser     999  4096 Oct  8 13:42 ..
-rw-------  1 chmuser     999    80 Jun 23  2015 admin-keyfile
-rw-------  1 chmuser     999 82078 Jun 23  2015 cacerts.jks
-rw-------  1 chmuser     999 50441 Jun 23  2015 default-web.xml
-rw-------  1 chmuser     999    32 Jun 23  2015 domain-passwords
-rw-r--r--  1 chmuser chmuser 28060 Oct  8 13:43 domain.xml
-rw-r--r--  1 chmuser chmuser 28059 Oct  8 13:43 domain.xml.bak
-rw-------  1 chmuser     999  2003 Jun 23  2015 keyfile
-rw-------  1 chmuser     999  4568 Jun 23  2015 keystore.jks
-rw-------  1 chmuser chmuser    41 Oct  8 13:42 local-password
-rw-r--r--  1 chmuser chmuser     0 Oct  8 13:42 lockfile
-rw-------  1 chmuser     999  5929 Jun 23  2015 logging.properties
-rw-------  1 chmuser     999  2497 Jun 23  2015 login.conf
-rw-r--r--  1 chmuser chmuser     2 Oct  8 13:42 pid
-rw-r--r--  1 chmuser chmuser     2 Oct  8 13:42 pid.prev
-rw-------  1 chmuser     999  6531 Jun 23  2015 server.policy
-rw-------  1 chmuser     999  3838 Jun 23  2015 sun-acc.xml
-rw-------  1 chmuser     999  7401 Jun 23  2015 wss-server-config-1.0.xml
-rw-------  1 chmuser     999  7907 Jun 23  2015 wss-server-config-2.0.xml
 ---> b90b0f27e6c7
Removing intermediate container 8d6962d314df
Step 3 : RUN id
 ---> Running in 5879ba94c68e
uid=999(chmuser) gid=1001(chmuser) groups=1001(chmuser)
 ---> 01b93b71c4f0
Removing intermediate container 5879ba94c68e
Step 4 : RUN touch /opt/glassfish3/glassfish/domains/domain1/config/xxxx
 ---> Running in b69312f22dd0
touch: cannot touch '/opt/glassfish3/glassfish/domains/domain1/config/xxxx': Permission denied
The command '/bin/sh -c touch /opt/glassfish3/glassfish/domains/domain1/config/xxxx' returned a non-zero code: 1

Running it on my own workstation works as expected. My workstation is using the overlay driver. Docker info:

Containers: 18
 Running: 7
 Paused: 0
 Stopped: 11
Images: 938
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.6-040406-lowlatency
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.54 GiB
Name: vor
ID: KAMR:T67O:SN4G:5EPS:VD77:KXY6:ID3P:LNPG:P2PF:F7RF:4OJD:HE7B

amon-ra pushed a commit to oondeo/docker-odoo that referenced this issue Apr 9, 2016
@fflatorre
Copy link

fflatorre commented Jun 3, 2016

Hi,

I've been beaten by this bug few hours ago.

I can confirm the behaviour : get broen permissions (?????) until you first "access" the resource as root.
As stated by someone else a simple "ls" as root first will fix everything.
I've actually added the following to my entrypoint logic :

ls -laR /

Env info :

$ sudo docker --version
Docker version 1.11.1, build 5604cbe
$ sudo docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 169
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /space/docker/aufs
 Backing Filesystem: extfs
 Dirs: 416
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.16.0-71-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.41 GiB
Name: lp01696
ID: KJ3L:FMBL:L2TX:PR6V:R3KK:OJR3:L4B2:UQEH:TZP4:QICF:KI33:3X53
Docker Root Dir: /space/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
$ uname -a
Linux ------- 3.16.0-71-generic #92~14.04.1-Ubuntu SMP Thu May 12 23:31:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:        14.04
Codename:       trusty

Cheers,
Francesco

@TactfulElf
Copy link

TactfulElf commented Aug 23, 2016

Hi,

I noticed and fixed this for myself last year but hadn't got around to reporting it.. :(
(I've dropped a note to aufs-users at lists.sourceforge.net)

It appears that the current implementation of aufs's dirperm1 doesn't use the correct permission overriding method when checking for whiteouts in dentry.c

The below bash script can simulate the issue:

#!/bin/bash

BASE=/root/aufsissue
BASE2=$BASE/aufs

#Create testuser
#adduser testissue

#Cleanup previous executions
umount $BASE2/mnt 2>/dev/null
rm -R $BASE2 2>/dev/null

#Setup a mount directory and two branches to be overlaid
mkdir $BASE2 $BASE2/branch1 $BASE2/branch2 $BASE2/mnt

#Mount branch1 on $BASE2/mnt
mount -t aufs -o dirperm1,dio,br:$BASE2/branch1=rw none $BASE2/mnt

#make two test directories on the lower branch
mkdir $BASE2/mnt/test1 $BASE2/mnt/test2

#Change permissions of two test directories
chmod -R 755 $BASE2/mnt/test1
chmod -R 700  $BASE2/mnt/test2

#Mount branch2 over branch1 in $BASE2/mnt
mount -t aufs -o remount,dirperm1,mod:$BASE2/branch1=ro+wh,prepend:$BASE2/branch2=rw none $BASE2/mnt

#Change ownership of two test directories
chown -R testissue:testissue $BASE2/mnt/test1
chown -R testissue:testissue $BASE2/mnt/test2

#Works correctly
echo
echo This works correctly
su testissue -c "touch $BASE2/mnt/test1/works"
#ls shows file was created
ls -lart $BASE2/branch2/test1

#Fails as unable to check for whiteouts with au_wh_test
#Even though the user "testissue" has ownership it cant read whiteouts 
#from the lower branch with permission 700 because at the lower branch testissue isnt the owner.
#change dentry.c
#wh_found = au_wh_test(h_parent, wh_name, /*try_sio*/0);
#to
#wh_found = au_wh_test(h_parent, wh_name, /*try_sio*/ignore_perm);
echo
echo This fails
su testissue -c "touch $BASE2/mnt/test2/fails"
#ls shows no file was created
ls -lart $BASE2/branch2/test2

I wrote the patch below for aufs and that fixed the issue for me - (it meant some kernel recompiling though):

--- a/fs/aufs/dentry.c
+++ b/fs/aufs/dentry.c
@@ -57,7 +57,7 @@
    br = au_sbr(dentry->d_sb, bindex);
    wh_able = !!au_br_whable(br->br_perm);
    if (wh_able)
-       wh_found = au_wh_test(h_parent, wh_name, /*try_sio*/0);
+       wh_found = au_wh_test(h_parent, wh_name, ignore_perm);
    h_dentry = ERR_PTR(wh_found);
    if (!wh_found)
        goto real_lookup;

Regards

Dave

@otherjason
Copy link

@TactfulElf That looks promising. Have you contributed the patch to the aufs project? That may be the best way of getting this fixed.

@TactfulElf
Copy link

@otherjason I've emailed aufs-users at lists.sourceforge.net hopefully someone will be able to review.

@TactfulElf
Copy link

TactfulElf commented Aug 31, 2016

Hi,

I had an update from the aufs developer:

Hello Dave,

Thanx for the report and the patch.
You are right. That is a bug of aufs dirperm1 option. I will include
your patch into next release. I hope you won't mind that the fix won't
be merged into aufs3 and aufs4.0.

J. R. Okajima

By the end of this month (today), aufs3.1[89] and aufs4.0 are not
maintained anymore. My new development base is aufs4.1.

J. R. Okajima

@TactfulElf
Copy link

TactfulElf commented Sep 5, 2016

patch has been added to aufs >=4.(x>=1)-20160905

If you have an older aufs version <4.1 because of an older kernel I guess the only option is to apply the patch manually.

sfjro/aufs4-standalone@625634d#diff-766898e31d29972e93ec410af8228d9a

@AkihiroSuda
Copy link
Member

Is this issue closable?
PTAL @thaJeztah

@thaJeztah
Copy link
Member

Looks like we should close this, yes, as it's not a bug in docker, and fixed upstream

@snth
Copy link

snth commented Nov 14, 2016

I was also affected by this bug in aufs. Switching to overlay fixed it but it reappears in overlay2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kernel area/storage/aufs kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
Projects
None yet
Development

No branches or pull requests

10 participants