readlink /proc/self/exe permission denied inside created thread of container entry point

[davidnoor]

Docker Version:

dnoor@gears:~/.../amps-docker/bug$ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:08:45 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:08:45 UTC 2015
 OS/Arch:      linux/amd64

Docker info

dnoor@gears:~/.../amps-docker/bug$ docker info
Containers: 36
Images: 112
Storage Driver: devicemapper
 Pool Name: docker-253:0-204068894-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 8.538 GB
 Data Space Total: 107.4 GB
 Data Space Available: 34.78 GB
 Metadata Space Used: 10.62 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.137 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 48
Total Memory: 125.6 GiB
Name: gears
ID: YMWT:GHAN:CDOK:L4B5:PNLC:JHWP:2APM:T3LC:HEFL:QSWJ:UG4W:ZSOC

uname -a

dnoor@gears:~/.../amps-docker/bug$ uname -a
Linux gears 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details: physical box

The results I received:
Calls to readlink() on anything that's a link inside of /proc/self return permission denied, for my entrypoint process in a docker container (using the ubuntu container as a base), and if I'm calling it from a thread other than the main thread.

The results I expected:
Calls to readlink() on valid links inside /proc/self work correctly from any thread in my process, even if my process is a docker entrypoint.

Additional info I think is important:
Here's a simple test program that uses /proc/self/exe:

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <sys/syscall.h>

int gettid()
{
  return syscall(SYS_gettid);
}
void* check_proc_self_exe(void* unused)
{
  const char *path = "/proc/self/exe";
  char outputBuffer[256];

  while(1)
  {
    memset(outputBuffer,0,sizeof(outputBuffer));
    if(readlink(path,outputBuffer,sizeof(outputBuffer))==-1)
      fprintf(stderr,"pid %d tid %d path [%s] readlink error [%s]\n", getpid(),gettid(),path,strerror(errno));
    else
      fprintf(stderr,"pid %d tid %d path [%s] readlink returned [%s]\n", getpid(),gettid(),path,outputBuffer);
    sleep(1);
  }

  return NULL;
}


int main()
{
  pthread_t thread;
  pthread_create(&thread,NULL,check_proc_self_exe,NULL);
  /* never returns */
  check_proc_self_exe(NULL);
  return 0;
}

Here's how I compile and run it on my host (centos 7)

gcc procself_bug.c -lpthread && docker run -v `pwd`:/host --entrypoint /host/a.out ubuntu

Here's what I see as output:

pid 1 tid 6 path [/proc/self/exe] readlink error [Permission denied]
pid 1 tid 1 path [/proc/self/exe] readlink returned [/host/a.out]
...

If I just run the program on my host, or anywhere else, or even with docker run -ti -v``pwd``:/host --entrypoint /bin/bash ubuntu -c /host/a.out, it works:

pid 1 tid 6 path [/proc/self/exe] readlink returned [/host/a.out]
pid 1 tid 1 path [/proc/self/exe] readlink returned [/host/a.out]
...

Thanks, -David

[davidnoor]

Thanks, I've updated the bug report above; happy to answer any additional questions.

[thaJeztah]

I found these issues, not sure at all if they're related (haven't looked in-depth), but adding them here in case they are; #8244 #11462

[davidnoor]

Thanks @thaJeztah . We saw #11462 as well and wondered if it might be related. It should be noted that I also see permission denied when calling readlink() on /proc/self/fd/* -- not just for the ttys, but even for files my process (in my "real" program -- not this test program) opens. I thought boiling it down to this simple test case with /proc/self/exe might help debugging though.

[justincormack]

I believe this is fixed in newer kernels where there is a "same thread group" check not just a same pid check

http://lxr.free-electrons.com/source/fs/proc/fd.c#L304

I am not sure exactly when this changed. (Found this issue while searching for a probably unrelated thing!)

[justincormack]

Going to close this as I believe it is fixed. Please comment or reopen if this is not the case.

[Vanuan]

It looks like the issue I have in MariaDB under CentOS 7.4:
https://bugs.alpinelinux.org/issues/7345

So when mysql client queries SHOW FULL FIELDS FROM mysqld creates a new thread in which it writes to /tmp/#sql_9_0.MAI and uses readlink to read it back:

readlink("/proc/self/fd/81", 0x7f880910abb0, 4095) = -1 EACCES (Permission denied)

The container starts as root, but mysqld uses setuid(100) to change it to other user. It also uses a separate thread for each connected client.

The kernel version is 3.10.0-693.2.2.el7.x86_64.

Does it look related?

[Vanuan]

Speculative findings:
https://stackoverflow.com/a/48178214/99024

[Vanuan]

To illustrate the difference better:

Good (v4.14):
https://github.com/torvalds/linux/blob/bebc6082da0a9f5d47a1ea2edc099bf671058bd4/fs/proc/fd.c#L283-L305

Bad (v3.10):
https://github.com/torvalds/linux/blob/8bb495e3f02401ee6f76d1b1d77f3ac9f079e376/fs/proc/fd.c#L303-L317

[Vanuan]

The earliest version of linux kernel the fix is included in is 4.4
It looks like the original security fix causing issues was backported to RHEL's kernel 3.10. But the next regression fix wasn't backported?

[raarts]

@Vanuan did you make any progress on this? I have exactly this problem with opendkim in an Alpine container running on CentOS 7.

[Vanuan]

@raarts I've outlined solutions and workarounds in the stackoverflow answer above:

• upgrade kernel
• convince musl maintainers to change realpath implementation and upgrade to that version
• don't use musl or any other code that relies on this /proc behavior
• downgrade kernel
• apply kernel patch
• convince CentOS maintainers to apply kernel patch
• don't use setuid (run containers under root)

[raarts]

@Vanuan, just hoping you put in a ticket in redhat's bugzilla. Thanks for tracking this down though. For the moment I dropped alpine for this container.