New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker build should support privileged operations #1916

Open
darklajid opened this Issue Sep 18, 2013 · 276 comments

Comments

Projects
None yet
@darklajid

Currently there seems to be no way to run privileged operations outside of docker run -privileged.

That means that I cannot do the same things in a Dockerfile. My recent issue: I'd like to run fuse (for encfs) inside of a container. Installing fuse is already a mess with hacks and ugly workarounds (see [1] and [2]), because mknod fails/isn't supported without a privileged build step.

The only workaround right now is to do the installation manually, using run -privileged, and creating a new 'fuse base image'. Which means that I cannot describe the whole container, from an official base image to finish, in a single Dockerfile.

I'd therefor suggest adding either

  • a docker build -privileged
    this should do the same thing as run -privileged, i.e. removing all caps limitations

or

  • a RUNP command in the Dockerfile
    this should .. well .. RUN, but with _P_rivileges

I tried looking at the source, but I'm useless with go and couldn't find a decent entrypoint to attach a proof of concept, unfortunately. :(

1: https://github.com/rogaha/docker-desktop/blob/master/Dockerfile#L40
2: #514 (comment)

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Sep 18, 2013

Collaborator

If we go for this, I'm more in favor of the RUNP option, instead of having
all container running in -privileged mode.

On Wed, Sep 18, 2013 at 1:07 PM, Benjamin Podszun
notifications@github.comwrote:

Currently there seems to be no way to run privileged operations outside of
docker run -privileged.

That means that I cannot do the same things in a Dockerfile. My recent
issue: I'd like to run fuse (for encfs) inside of a container. Installing
fuse is already a mess with hacks and ugly workarounds (see [1] and [2]),
because mknod fails/isn't supported without a privileged build step.

The only workaround right now is to do the installation manually, using
run -privileged, and creating a new 'fuse base image'. Which means that I
cannot describe the whole container, from an official base image to finish,
in a single Dockerfile.

I'd therefor suggest adding either

  • a docker build -privileged
    this should do the same thing as run -privileged, i.e. removing all
    caps limitations

or

  • a RUNP command in the Dockerfile
    this should .. well .. RUN, but with _P_rivileges

I tried looking at the source, but I'm useless with go and couldn't find a
decent entrypoint to attach a proof of concept, unfortunately. :(

1: https://github.com/rogaha/docker-desktop/blob/master/Dockerfile#L40
2: #514 (comment)#514 (comment)


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/1916
.

Victor VIEUX
http://vvieux.com

Collaborator

vieux commented Sep 18, 2013

If we go for this, I'm more in favor of the RUNP option, instead of having
all container running in -privileged mode.

On Wed, Sep 18, 2013 at 1:07 PM, Benjamin Podszun
notifications@github.comwrote:

Currently there seems to be no way to run privileged operations outside of
docker run -privileged.

That means that I cannot do the same things in a Dockerfile. My recent
issue: I'd like to run fuse (for encfs) inside of a container. Installing
fuse is already a mess with hacks and ugly workarounds (see [1] and [2]),
because mknod fails/isn't supported without a privileged build step.

The only workaround right now is to do the installation manually, using
run -privileged, and creating a new 'fuse base image'. Which means that I
cannot describe the whole container, from an official base image to finish,
in a single Dockerfile.

I'd therefor suggest adding either

  • a docker build -privileged
    this should do the same thing as run -privileged, i.e. removing all
    caps limitations

or

  • a RUNP command in the Dockerfile
    this should .. well .. RUN, but with _P_rivileges

I tried looking at the source, but I'm useless with go and couldn't find a
decent entrypoint to attach a proof of concept, unfortunately. :(

1: https://github.com/rogaha/docker-desktop/blob/master/Dockerfile#L40
2: #514 (comment)#514 (comment)


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/1916
.

Victor VIEUX
http://vvieux.com

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Sep 19, 2013

Contributor

Actually, we might have to do both — i.e., RUNP + require a "-privileged"
flag.

If we rely only on RUNP (without requiring "-privileged"), then we would
have to wonder when we do a build "is this build safe?".
If we rely only on "-privileged", we miss the information (in the
Dockerfile) that "this action requires extended privileges".

I think a combination of both is the safest way.

On Wed, Sep 18, 2013 at 4:07 AM, Benjamin Podszun
notifications@github.comwrote:

Currently there seems to be no way to run privileged operations outside of
docker run -privileged.

That means that I cannot do the same things in a Dockerfile. My recent
issue: I'd like to run fuse (for encfs) inside of a container. Installing
fuse is already a mess with hacks and ugly workarounds (see [1] and [2]),
because mknod fails/isn't supported without a privileged build step.

The only workaround right now is to do the installation manually, using
run -privileged, and creating a new 'fuse base image'. Which means that I
cannot describe the whole container, from an official base image to finish,
in a single Dockerfile.

I'd therefor suggest adding either

  • a docker build -privileged
    this should do the same thing as run -privileged, i.e. removing all
    caps limitations

or

  • a RUNP command in the Dockerfile
    this should .. well .. RUN, but with _P_rivileges

I tried looking at the source, but I'm useless with go and couldn't find a
decent entrypoint to attach a proof of concept, unfortunately. :(

1: https://github.com/rogaha/docker-desktop/blob/master/Dockerfile#L40
2: #514 (comment)#514 (comment)


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/1916
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

Contributor

jpetazzo commented Sep 19, 2013

Actually, we might have to do both — i.e., RUNP + require a "-privileged"
flag.

If we rely only on RUNP (without requiring "-privileged"), then we would
have to wonder when we do a build "is this build safe?".
If we rely only on "-privileged", we miss the information (in the
Dockerfile) that "this action requires extended privileges".

I think a combination of both is the safest way.

On Wed, Sep 18, 2013 at 4:07 AM, Benjamin Podszun
notifications@github.comwrote:

Currently there seems to be no way to run privileged operations outside of
docker run -privileged.

That means that I cannot do the same things in a Dockerfile. My recent
issue: I'd like to run fuse (for encfs) inside of a container. Installing
fuse is already a mess with hacks and ugly workarounds (see [1] and [2]),
because mknod fails/isn't supported without a privileged build step.

The only workaround right now is to do the installation manually, using
run -privileged, and creating a new 'fuse base image'. Which means that I
cannot describe the whole container, from an official base image to finish,
in a single Dockerfile.

I'd therefor suggest adding either

  • a docker build -privileged
    this should do the same thing as run -privileged, i.e. removing all
    caps limitations

or

  • a RUNP command in the Dockerfile
    this should .. well .. RUN, but with _P_rivileges

I tried looking at the source, but I'm useless with go and couldn't find a
decent entrypoint to attach a proof of concept, unfortunately. :(

1: https://github.com/rogaha/docker-desktop/blob/master/Dockerfile#L40
2: #514 (comment)#514 (comment)


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/1916
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

@darklajid

This comment has been minimized.

Show comment
Hide comment
@darklajid

darklajid Sep 19, 2013

Sounds reasonable. For me this feature (being able to create device nodes) makes or breaks the ability to create the deployment in Docker. If I can help (testing mostly, I tried looking at the source but failed so far. It seems the available commands in a buildfile are found via reflection, I added a runp command that set the config.privileged to true, but so far I'm unable to build and test -> stuck) I'd gladly invest some time.

Sounds reasonable. For me this feature (being able to create device nodes) makes or breaks the ability to create the deployment in Docker. If I can help (testing mostly, I tried looking at the source but failed so far. It seems the available commands in a buildfile are found via reflection, I added a runp command that set the config.privileged to true, but so far I'm unable to build and test -> stuck) I'd gladly invest some time.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Sep 20, 2013

Contributor

I'd suggest RUNP + build -privileged.

lights up some smoke signals to catch attention of @shykes, @crosbymichael

... And then we'll have to find someone to implement it, of course ☺
Would that be something you'd want to try (with appropriate guidance and feedback from the core team, of course) ?

Contributor

jpetazzo commented Sep 20, 2013

I'd suggest RUNP + build -privileged.

lights up some smoke signals to catch attention of @shykes, @crosbymichael

... And then we'll have to find someone to implement it, of course ☺
Would that be something you'd want to try (with appropriate guidance and feedback from the core team, of course) ?

@darklajid

This comment has been minimized.

Show comment
Hide comment
@darklajid

darklajid Sep 20, 2013

If the last part was targeted at me: Sure, why not. I'm already messing with the go code (not a language I'm familiar with, but see above: I'm trying to figure out what's going on anyway).

With a couple of pointers / someone to ping for some questions I'd certainly give it a try.

If the last part was targeted at me: Sure, why not. I'm already messing with the go code (not a language I'm familiar with, but see above: I'm trying to figure out what's going on anyway).

With a couple of pointers / someone to ping for some questions I'd certainly give it a try.

@shykes

This comment has been minimized.

Show comment
Hide comment
@shykes

shykes Sep 20, 2013

Collaborator

I'm not sold on RUNP or build -privileged.

​Generally I don't like anything that introduces different possible builds of the same input. That's why you can't pass arguments or env variables to a build.

Specifically I don't like introducing dependencies on "privileged" all over the place, because it designates a set of capabilities that is a) very large and b) not clearly spec-ed or defined. That's ok as a coarse mechanism for sysadmins to bypass security in an all-or-nothing way - an "escape hatch" when the standard docker execution environment is not enough. It's similar in that way to bind-mounts and custom lxc-conf.


@solomonstre
@getdocker

On Fri, Sep 20, 2013 at 3:18 PM, Benjamin Podszun
notifications@github.com wrote:

If the last part was targeted at me: Sure, why not. I'm already messing with the go code (not a language I'm familiar with, but see above: I'm trying to figure out what's going on anyway).

With a couple of pointers / someone to ping for some questions I'd certainly give it a try.

Reply to this email directly or view it on GitHub:
#1916 (comment)

Collaborator

shykes commented Sep 20, 2013

I'm not sold on RUNP or build -privileged.

​Generally I don't like anything that introduces different possible builds of the same input. That's why you can't pass arguments or env variables to a build.

Specifically I don't like introducing dependencies on "privileged" all over the place, because it designates a set of capabilities that is a) very large and b) not clearly spec-ed or defined. That's ok as a coarse mechanism for sysadmins to bypass security in an all-or-nothing way - an "escape hatch" when the standard docker execution environment is not enough. It's similar in that way to bind-mounts and custom lxc-conf.


@solomonstre
@getdocker

On Fri, Sep 20, 2013 at 3:18 PM, Benjamin Podszun
notifications@github.com wrote:

If the last part was targeted at me: Sure, why not. I'm already messing with the go code (not a language I'm familiar with, but see above: I'm trying to figure out what's going on anyway).

With a couple of pointers / someone to ping for some questions I'd certainly give it a try.

Reply to this email directly or view it on GitHub:
#1916 (comment)

@darklajid

This comment has been minimized.

Show comment
Hide comment
@darklajid

darklajid Sep 21, 2013

Well, do you agree that it should be possible to build a docker image that - for example - runs fuse?
For that we'd need to mknod.

The way I see it, there's no way these builds could be different depending on parameters: The build will work (caps are not / less restricted than now) or fail (status quo). There's little to no risk of different 'versions' of the same build file, right?

Well, do you agree that it should be possible to build a docker image that - for example - runs fuse?
For that we'd need to mknod.

The way I see it, there's no way these builds could be different depending on parameters: The build will work (caps are not / less restricted than now) or fail (status quo). There's little to no risk of different 'versions' of the same build file, right?

@lukewpatterson

This comment has been minimized.

Show comment
Hide comment
@lukewpatterson

lukewpatterson Sep 23, 2013

I'm running into this issue now. To build the image I need, I have to perform a series of run -privileged steps + a commit step, rather than building a Dockerfile. Ideally, it would be nice to express the image build steps in a Dockerfile.

I'm running into this issue now. To build the image I need, I have to perform a series of run -privileged steps + a commit step, rather than building a Dockerfile. Ideally, it would be nice to express the image build steps in a Dockerfile.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Sep 23, 2013

Contributor

Is it also related to mknod operations?
If you could describe exactly the actions that require privileged mode in
your case, it would be very helpful!
Thanks,

Contributor

jpetazzo commented Sep 23, 2013

Is it also related to mknod operations?
If you could describe exactly the actions that require privileged mode in
your case, it would be very helpful!
Thanks,

@lukewpatterson

This comment has been minimized.

Show comment
Hide comment
@lukewpatterson

lukewpatterson Sep 23, 2013

Hey @jpetazzo, from the mailing list, here is the issue I'm facing: https://groups.google.com/forum/#!topic/docker-user/1pFhqlfbqQI

I'm trying to mount a fs I created (created to work around aufs and something about journaling) inside the container. The specific command I'm running is mount -o loop=/dev/loop0 /db/disk-image /home/db2inst1, where /db/disk-image was created with dd if=/dev/zero of=disk-image count=409600 and home/db2inst1 is where I'm trying to start db2 from.

Hey @jpetazzo, from the mailing list, here is the issue I'm facing: https://groups.google.com/forum/#!topic/docker-user/1pFhqlfbqQI

I'm trying to mount a fs I created (created to work around aufs and something about journaling) inside the container. The specific command I'm running is mount -o loop=/dev/loop0 /db/disk-image /home/db2inst1, where /db/disk-image was created with dd if=/dev/zero of=disk-image count=409600 and home/db2inst1 is where I'm trying to start db2 from.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Sep 23, 2013

Contributor

If I understand correctly, during the installation process, you need a non-AUFS directory — or rather, something that supports O_DIRECT. If that's the case, Docker 0.7 should solve the problem, since it will use ext4 (and block-level snapshots) instead of AUFS.

Contributor

jpetazzo commented Sep 23, 2013

If I understand correctly, during the installation process, you need a non-AUFS directory — or rather, something that supports O_DIRECT. If that's the case, Docker 0.7 should solve the problem, since it will use ext4 (and block-level snapshots) instead of AUFS.

@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 2, 2013

+1 for this as well.

Installing packages that require change to memory settings and kernel configuration (e.g. Vertica DB, WebSphere MQ) can only be done in privileged mode.

+1 for this as well.

Installing packages that require change to memory settings and kernel configuration (e.g. Vertica DB, WebSphere MQ) can only be done in privileged mode.

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Oct 2, 2013

Contributor

Let's try to separate concerns when it comes to running / building with "privileged": it can be required just during the build, just during execution via docker run or both.

It should be possible to allow a build to do something requiring a bit more permissions for a step (or more) if that's necessary. I needed this for a project and had to convert half of a Dockerfile to a shell script which invoked the build and continued to build things in privileged mode, so having a "privileged" build would be useful.

However, we shouldn't go all the way down to privileged mode by default just so we can use sysctl to change some settings. This should be done via image configuration or via command line args to be passed to docker run.

Contributor

unclejack commented Oct 2, 2013

Let's try to separate concerns when it comes to running / building with "privileged": it can be required just during the build, just during execution via docker run or both.

It should be possible to allow a build to do something requiring a bit more permissions for a step (or more) if that's necessary. I needed this for a project and had to convert half of a Dockerfile to a shell script which invoked the build and continued to build things in privileged mode, so having a "privileged" build would be useful.

However, we shouldn't go all the way down to privileged mode by default just so we can use sysctl to change some settings. This should be done via image configuration or via command line args to be passed to docker run.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 3, 2013

Contributor

Right. @orikremer, do you have details on the parameters that Vertica DB and WebSphere MQ were trying to change?

If it's stuff in /sys or /proc, the best solution might be to put some mock up there instead, rather than switching to privileged mode (since the changes won't be persisted anyway).

In the long run, a mock filesystem might capture the change and convert them to Dockerfile directives, instructing the runtime that "hey, this container needs such or such tweak".

Contributor

jpetazzo commented Oct 3, 2013

Right. @orikremer, do you have details on the parameters that Vertica DB and WebSphere MQ were trying to change?

If it's stuff in /sys or /proc, the best solution might be to put some mock up there instead, rather than switching to privileged mode (since the changes won't be persisted anyway).

In the long run, a mock filesystem might capture the change and convert them to Dockerfile directives, instructing the runtime that "hey, this container needs such or such tweak".

@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 3, 2013

@jpetazzo It's been a couple of days since I created the image. AFAIR Vertica was complaining that it doesn't have enough memory and both were trying to change max open files.
I will try to recreate the image using a Dockerfile and report back.

@jpetazzo It's been a couple of days since I created the image. AFAIR Vertica was complaining that it doesn't have enough memory and both were trying to change max open files.
I will try to recreate the image using a Dockerfile and report back.

@ewindisch

This comment has been minimized.

Show comment
Hide comment
@ewindisch

ewindisch Oct 3, 2013

Contributor

Noting issue #2080 as it is related.

Contributor

ewindisch commented Oct 3, 2013

Noting issue #2080 as it is related.

@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 6, 2013

@jpetazzo started recreating the image without -privileged. Two issues so far:

  • nice in limits.conf: Vertica adds "dbadmin - nice 0" to /etc/security/limits.conf. When trying to switch to that user when running in a non-privileged container I get a "could not open session" error. In a privileged container switch user works with no errors.
  • max open files: since the max needed in the container was higher than the one set in host I had to change /etc/init/docker.conf on the host and set "limit nofile" and then ulimit -n in the container. Is that the correct approach ?

@jpetazzo started recreating the image without -privileged. Two issues so far:

  • nice in limits.conf: Vertica adds "dbadmin - nice 0" to /etc/security/limits.conf. When trying to switch to that user when running in a non-privileged container I get a "could not open session" error. In a privileged container switch user works with no errors.
  • max open files: since the max needed in the container was higher than the one set in host I had to change /etc/init/docker.conf on the host and set "limit nofile" and then ulimit -n in the container. Is that the correct approach ?
@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 6, 2013

Contributor

When trying to switch to that user,

How does the switch happen? I don't understand how -privileged would help with user-switching; I'm probably missing something here :-)

max open files

If I understand correctly, the Vertical installer tries to set the max number of open files to a very high number, and that only works if Docker was started with such a high number or with the -privileged flag; right?

Contributor

jpetazzo commented Oct 6, 2013

When trying to switch to that user,

How does the switch happen? I don't understand how -privileged would help with user-switching; I'm probably missing something here :-)

max open files

If I understand correctly, the Vertical installer tries to set the max number of open files to a very high number, and that only works if Docker was started with such a high number or with the -privileged flag; right?

@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 7, 2013

switching user - su dbadmin fails with that error.
I was able to reproduce by:

  • pull a new image (centos-6.4-x86_64) and run non privileged
  • useradd testuser
  • edit /etc/security/limits.conf, add "testuser - nice 0"
  • try su testuser --> should fail with "could not open session"
    In a -privileged container su testuser works fine.

max open files - correct. the installer tries to set to a number higher than the host has. Only by increasing the hosts setting or starting -privileged does this work.

switching user - su dbadmin fails with that error.
I was able to reproduce by:

  • pull a new image (centos-6.4-x86_64) and run non privileged
  • useradd testuser
  • edit /etc/security/limits.conf, add "testuser - nice 0"
  • try su testuser --> should fail with "could not open session"
    In a -privileged container su testuser works fine.

max open files - correct. the installer tries to set to a number higher than the host has. Only by increasing the hosts setting or starting -privileged does this work.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 7, 2013

Contributor

I just tried with the following Dockerfile:

FROM ubuntu
RUN useradd testuser
RUN echo testuser - nice 0 > /etc/security/limits.conf
CMD su testuser

And it works fine. What's the exact name of the image you're using?
(I tried centos-6.4-x86_64 but looks like I can't pull it!)

Contributor

jpetazzo commented Oct 7, 2013

I just tried with the following Dockerfile:

FROM ubuntu
RUN useradd testuser
RUN echo testuser - nice 0 > /etc/security/limits.conf
CMD su testuser

And it works fine. What's the exact name of the image you're using?
(I tried centos-6.4-x86_64 but looks like I can't pull it!)

@mikewaters

This comment has been minimized.

Show comment
Hide comment
@mikewaters

mikewaters Oct 7, 2013

@lukewpatterson Can you share how you got the loop filesystem working inside your container?

@lukewpatterson Can you share how you got the loop filesystem working inside your container?

@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 7, 2013

@jpetazzo Running this docker file

FROM backjlack/centos-6.4-x86_64
RUN useradd testuser
RUN echo 'testuser - nice 0' >> /etc/security/limits.conf
RUN su testuser
RUN echo 'test' > ~/test.txt

failed with:

ori@ubuntu:~/su_test$ sudo docker build .
Uploading context 10240 bytes
Step 1 : FROM backjlack/centos-6.4-x86_64
 ---> b1343935b9e5
Step 2 : RUN useradd testuser
 ---> Running in b41d9aa2be1b
 ---> 2ff05b54e806
Step 3 : RUN echo 'testuser - nice 0' >> /etc/security/limits.conf
 ---> Running in e83291fafc66
 ---> 03b85baf140a
Step 4 : RUN su testuser
 ---> Running in c289f6e5f3f4
could not open session
Error build: The command [/bin/sh -c su testuser] returned a non-zero code: 1
The command [/bin/sh -c su testuser] returned a non-zero code: 1
ori@ubuntu:~/su_test$

@jpetazzo Running this docker file

FROM backjlack/centos-6.4-x86_64
RUN useradd testuser
RUN echo 'testuser - nice 0' >> /etc/security/limits.conf
RUN su testuser
RUN echo 'test' > ~/test.txt

failed with:

ori@ubuntu:~/su_test$ sudo docker build .
Uploading context 10240 bytes
Step 1 : FROM backjlack/centos-6.4-x86_64
 ---> b1343935b9e5
Step 2 : RUN useradd testuser
 ---> Running in b41d9aa2be1b
 ---> 2ff05b54e806
Step 3 : RUN echo 'testuser - nice 0' >> /etc/security/limits.conf
 ---> Running in e83291fafc66
 ---> 03b85baf140a
Step 4 : RUN su testuser
 ---> Running in c289f6e5f3f4
could not open session
Error build: The command [/bin/sh -c su testuser] returned a non-zero code: 1
The command [/bin/sh -c su testuser] returned a non-zero code: 1
ori@ubuntu:~/su_test$
@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 7, 2013

Contributor

I turned on debugging for the PAM module (by adding debug to the pam_limits.so line in /etc/pam.d/system-auth), installed syslog, tried to su again, and here's what I found in /var/log/secure:

Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): reading settings from '/etc/security/limits.conf'
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): process_limit: processing - nice 0 for USER
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): reading settings from '/etc/security/limits.d/90-nproc.conf'
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): process_limit: processing soft nproc 1024 for DEFAULT
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): Could not set limit for 'nice': Operation not permitted

Then I straced the su process, and found out that the following system call was failing:

setrlimit(RLIMIT_NICE, {rlim_cur=20, rlim_max=20}) = -1 EPERM (Operation not permitted)

This, in turn, causes the pam_limits module to report a failure; and this prevents su from continuing.
Interestingly, on Ubuntu, pam_limits is not enabled for su by default; and even if you enable it, the setrlimit call fails, but su continues and works anyway.
It might be related to the audit code, I'm not sure.

Now, why is setrlimit failing? Because the container is missing the sys_resource capability, which is required to raise any kind of limit.

I think I would just comment out that limits.conf directive.
(By the way, it's bad practice to add stuff directly to limits.conf; it should go to a separate file in limits.d, I think.)

Note: since you already increased the limit for number of open files for Docker, you could also raise the limit for the max priority; that should work as well!

I hope this helps.

Contributor

jpetazzo commented Oct 7, 2013

I turned on debugging for the PAM module (by adding debug to the pam_limits.so line in /etc/pam.d/system-auth), installed syslog, tried to su again, and here's what I found in /var/log/secure:

Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): reading settings from '/etc/security/limits.conf'
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): process_limit: processing - nice 0 for USER
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): reading settings from '/etc/security/limits.d/90-nproc.conf'
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): process_limit: processing soft nproc 1024 for DEFAULT
Oct 7 14:12:23 8be1e7bc5590 su: pam_limits(su:session): Could not set limit for 'nice': Operation not permitted

Then I straced the su process, and found out that the following system call was failing:

setrlimit(RLIMIT_NICE, {rlim_cur=20, rlim_max=20}) = -1 EPERM (Operation not permitted)

This, in turn, causes the pam_limits module to report a failure; and this prevents su from continuing.
Interestingly, on Ubuntu, pam_limits is not enabled for su by default; and even if you enable it, the setrlimit call fails, but su continues and works anyway.
It might be related to the audit code, I'm not sure.

Now, why is setrlimit failing? Because the container is missing the sys_resource capability, which is required to raise any kind of limit.

I think I would just comment out that limits.conf directive.
(By the way, it's bad practice to add stuff directly to limits.conf; it should go to a separate file in limits.d, I think.)

Note: since you already increased the limit for number of open files for Docker, you could also raise the limit for the max priority; that should work as well!

I hope this helps.

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Oct 7, 2013

Member

In that Dockerfile, you've got the following line by itself:

RUN su testuser

There's no command to go with this (and it won't apply any resulting shell to subsequent RUN commands), so I wouldn't be surprised if it's really failing at trying to open a shell and not being somewhere interactive that doing so makes sense (since docker build is not an interactive process). I don't have time right now to confirm, but it's probably worth a try with an actual command being passed to su.

Member

tianon commented Oct 7, 2013

In that Dockerfile, you've got the following line by itself:

RUN su testuser

There's no command to go with this (and it won't apply any resulting shell to subsequent RUN commands), so I wouldn't be surprised if it's really failing at trying to open a shell and not being somewhere interactive that doing so makes sense (since docker build is not an interactive process). I don't have time right now to confirm, but it's probably worth a try with an actual command being passed to su.

@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 7, 2013

@jpetazzo Thanks for the detailed description. I will try raising the max priority for Docker and see if that helps.

(By the way, it's bad practice to add stuff directly to limits.conf; it should go to a separate file in limits.d, I think.)

Agreed, but as this is the Vertica installer code that's putting it there I am trying work around that.

@tianon The same happens if I run this in an interactive shell (/bin/bash).

@jpetazzo Thanks for the detailed description. I will try raising the max priority for Docker and see if that helps.

(By the way, it's bad practice to add stuff directly to limits.conf; it should go to a separate file in limits.d, I think.)

Agreed, but as this is the Vertica installer code that's putting it there I am trying work around that.

@tianon The same happens if I run this in an interactive shell (/bin/bash).

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Oct 7, 2013

Member

My apologies, I think it was still worth a try.

The point about that line not making much sense in the Dockerfile still does apply though. You probably wanted something more like this (after you figure out the limits issues):

RUN su testuser -c 'echo test > ~/test.txt'
Member

tianon commented Oct 7, 2013

My apologies, I think it was still worth a try.

The point about that line not making much sense in the Dockerfile still does apply though. You probably wanted something more like this (after you figure out the limits issues):

RUN su testuser -c 'echo test > ~/test.txt'
@orikremer

This comment has been minimized.

Show comment
Hide comment
@orikremer

orikremer Oct 7, 2013

@tianon you're right, it doesn't make much sense. That was merely to demonstrate that the su itself fails.

@tianon you're right, it doesn't make much sense. That was merely to demonstrate that the su itself fails.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 10, 2013

Contributor

To get back to the original discussion: I believe it is fine from a security standpoint (and useful!) to allow setfcap and mknod capabilities in the build process (and probably in regular container execution as well). Does anyone see any problem that could stem from that?

Contributor

jpetazzo commented Oct 10, 2013

To get back to the original discussion: I believe it is fine from a security standpoint (and useful!) to allow setfcap and mknod capabilities in the build process (and probably in regular container execution as well). Does anyone see any problem that could stem from that?

@zigguratt

This comment has been minimized.

Show comment
Hide comment
@zigguratt

zigguratt Oct 13, 2013

@jpetazzo Quite the opposite! It would solve many problems I'm encountering. I think this is necessary for people who want to run Docker containers that act/look more like a real machine.

@jpetazzo Quite the opposite! It would solve many problems I'm encountering. I think this is necessary for people who want to run Docker containers that act/look more like a real machine.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 13, 2013

Contributor

OK! If that's fine with you, I'm closing this issue in favor of #2191, "Enable mknod and setfcap capabilities in all containers" then :-)

Contributor

jpetazzo commented Oct 13, 2013

OK! If that's fine with you, I'm closing this issue in favor of #2191, "Enable mknod and setfcap capabilities in all containers" then :-)

@jpetazzo jpetazzo closed this Oct 13, 2013

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 14, 2013

Contributor

Do we know about such scenarios?

On Sun, Oct 13, 2013 at 12:22 PM, unclejack notifications@github.comwrote:

#2191 #2191 doesn't solve this
problem for all scenarions which would require docker build -privileged.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/1916#issuecomment-26224788
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post:
http://jpetazzo.github.io/2013/10/06/policy-rc-d-do-not-start-services-automatically/

Contributor

jpetazzo commented Oct 14, 2013

Do we know about such scenarios?

On Sun, Oct 13, 2013 at 12:22 PM, unclejack notifications@github.comwrote:

#2191 #2191 doesn't solve this
problem for all scenarions which would require docker build -privileged.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/1916#issuecomment-26224788
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post:
http://jpetazzo.github.io/2013/10/06/policy-rc-d-do-not-start-services-automatically/

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Oct 15, 2013

Contributor

@jpetazzo This is needed when you want to use a Dockerfile to build an operating system image.

I've deleted my comment by mistake when editing it.

Please take a look at how this would look like without doing everything in a Dockerfile:

from ubuntu:12.04
run apt-get update
[... a few more run commands]
add build.sh /root/build.sh

build.sh

docker build -t mybuild .
docker run -i -t -privileged -cidfile mybuild.cid mybuild /root/build.sh
buildcid=`cat mybuild.cid`
rm mybuild.cid
docker commit $buildcid mybuild-final

This is just forcing me to work around the lack of runp in the Dockerfile or docker build -privileged, thus rendering Dockerfiles useless and forcing me write a tool to duplicate Dockerfile-like functionality.

Obviously, the Dockerfile would be much more useful with runp or docker build -privileged:
runp example:

from ubuntu:12.04
run apt-get update
[... a few more run commands]
add build.sh /root/build.sh
runp /root/build.sh

docker build -privileged example:

from ubuntu:12.04
run apt-get update
[... a few more run commands]
add build.sh /root/build.sh
run /root/build.sh
Contributor

unclejack commented Oct 15, 2013

@jpetazzo This is needed when you want to use a Dockerfile to build an operating system image.

I've deleted my comment by mistake when editing it.

Please take a look at how this would look like without doing everything in a Dockerfile:

from ubuntu:12.04
run apt-get update
[... a few more run commands]
add build.sh /root/build.sh

build.sh

docker build -t mybuild .
docker run -i -t -privileged -cidfile mybuild.cid mybuild /root/build.sh
buildcid=`cat mybuild.cid`
rm mybuild.cid
docker commit $buildcid mybuild-final

This is just forcing me to work around the lack of runp in the Dockerfile or docker build -privileged, thus rendering Dockerfiles useless and forcing me write a tool to duplicate Dockerfile-like functionality.

Obviously, the Dockerfile would be much more useful with runp or docker build -privileged:
runp example:

from ubuntu:12.04
run apt-get update
[... a few more run commands]
add build.sh /root/build.sh
runp /root/build.sh

docker build -privileged example:

from ubuntu:12.04
run apt-get update
[... a few more run commands]
add build.sh /root/build.sh
run /root/build.sh
@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 15, 2013

Contributor

@unclejack : sorry, my question was not accurate enough!
What I meant was "which permission do you need exactly (on top of mknod and setfcap)?"

Contributor

jpetazzo commented Oct 15, 2013

@unclejack : sorry, my question was not accurate enough!
What I meant was "which permission do you need exactly (on top of mknod and setfcap)?"

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Oct 15, 2013

Contributor

@jpetazzo It's difficult to say, I'd have to audit this somehow to figure out what's needed. Mounting file systems, using loopback mounted block devices and a few other things.

There are at least three separate needs when it comes to building images: permissions required during a docker build, permissions required when running a container with docker and runtime needs for processes during build, run or both (like sysctls and others).

I think having docker build -privileged (or runp to use -privileged mode only for the commands which really need it) would be useful.

Contributor

unclejack commented Oct 15, 2013

@jpetazzo It's difficult to say, I'd have to audit this somehow to figure out what's needed. Mounting file systems, using loopback mounted block devices and a few other things.

There are at least three separate needs when it comes to building images: permissions required during a docker build, permissions required when running a container with docker and runtime needs for processes during build, run or both (like sysctls and others).

I think having docker build -privileged (or runp to use -privileged mode only for the commands which really need it) would be useful.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 15, 2013

Contributor

Ah, mounts are definitely a big one. That's a very valid use-case, and we probably don't want to allow them in the general case. I'm re-opening the issue.

Contributor

jpetazzo commented Oct 15, 2013

Ah, mounts are definitely a big one. That's a very valid use-case, and we probably don't want to allow them in the general case. I'm re-opening the issue.

@jpetazzo jpetazzo reopened this Oct 15, 2013

@ramarnat

This comment has been minimized.

Show comment
Hide comment
@ramarnat

ramarnat Oct 17, 2013

@jpetazzo RE: PAM module (I am installing Vertica as well) would you suggest recompiling docker after taking the sys_resource out of the lxc.cap.drop?

Maybe some of these limits can be set via the docker.conf file?

@jpetazzo RE: PAM module (I am installing Vertica as well) would you suggest recompiling docker after taking the sys_resource out of the lxc.cap.drop?

Maybe some of these limits can be set via the docker.conf file?

@ewindisch

This comment has been minimized.

Show comment
Hide comment
@ewindisch

ewindisch Oct 17, 2013

Contributor

It should be considered that Docker itself might be running in a limited set of capabilities whereby these privileges may not be available for Docker to delegate to its containers. That would be especially true in a nested Docker scenario should issue #2080 land -- this might allow non-privileged nested Docker.

Not that this changes anything, except that solutions such as 'runp' or '-priviledged' might not be guaranteed success in all Docker environments. That should be considered when adding such commands and when documenting them.

Contributor

ewindisch commented Oct 17, 2013

It should be considered that Docker itself might be running in a limited set of capabilities whereby these privileges may not be available for Docker to delegate to its containers. That would be especially true in a nested Docker scenario should issue #2080 land -- this might allow non-privileged nested Docker.

Not that this changes anything, except that solutions such as 'runp' or '-priviledged' might not be guaranteed success in all Docker environments. That should be considered when adding such commands and when documenting them.

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Feb 16, 2017

Collaborator

@sneumann mybuilder would be an image name and have some CMD or ENTRYPOINT indeed. The contract for that workaround to work is that mybuilder will have to tar the context from within the container and let it go to stdout. That is passed on to docker build's stdin, thanks to the shell pipe | and is considered to be the context because the context path for docker build -t myimage - is -.

Collaborator

tiborvass commented Feb 16, 2017

@sneumann mybuilder would be an image name and have some CMD or ENTRYPOINT indeed. The contract for that workaround to work is that mybuilder will have to tar the context from within the container and let it go to stdout. That is passed on to docker build's stdin, thanks to the shell pipe | and is considered to be the context because the context path for docker build -t myimage - is -.

@mbana

This comment has been minimized.

Show comment
Hide comment
@mbana

mbana Feb 16, 2017

a bit strange upon looking at the code it seems that this option is avaialble in the build command:

anyone more clued up have any idea why it's not being applied?

mbana commented Feb 16, 2017

a bit strange upon looking at the code it seems that this option is avaialble in the build command:

anyone more clued up have any idea why it's not being applied?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 21, 2017

Member

@mbana the --security-opt is for native Windows containers, which support "credentialspec" #23389

Member

thaJeztah commented Feb 21, 2017

@mbana the --security-opt is for native Windows containers, which support "credentialspec" #23389

@mbana

This comment has been minimized.

Show comment
Hide comment
@mbana

mbana Feb 22, 2017

is it possible to modify this and make it persist so that future build will enable ptrace?

for anyone interested here are some good links:

mbana commented Feb 22, 2017

is it possible to modify this and make it persist so that future build will enable ptrace?

for anyone interested here are some good links:

@scjody

This comment has been minimized.

Show comment
Hide comment
@scjody

scjody Mar 23, 2017

I've seen a lot of claims from various folks that this feature isn't needed because builds can be changed to not require certain privileged operations, but no suggestions on what to do about the "docker in docker" case. If a build needs to run docker commands, for example to pull down some images that we want to ship inside this one, or to build a sub-image, how are we supposed to do this without some sort of privileged build option?

For now I'm going to work around this using docker run and docker commit but it would be great if docker build could support this use case.

scjody commented Mar 23, 2017

I've seen a lot of claims from various folks that this feature isn't needed because builds can be changed to not require certain privileged operations, but no suggestions on what to do about the "docker in docker" case. If a build needs to run docker commands, for example to pull down some images that we want to ship inside this one, or to build a sub-image, how are we supposed to do this without some sort of privileged build option?

For now I'm going to work around this using docker run and docker commit but it would be great if docker build could support this use case.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Mar 23, 2017

Contributor

@scjody Sounds like you want #31257

Contributor

cpuguy83 commented Mar 23, 2017

@scjody Sounds like you want #31257

@scjody

This comment has been minimized.

Show comment
Hide comment
@scjody

scjody Mar 23, 2017

@cpuguy83 I'm not sure that covers what's going on in this case but I'll give it a shot once that's merged. Thanks!

scjody commented Mar 23, 2017

@cpuguy83 I'm not sure that covers what's going on in this case but I'll give it a shot once that's merged. Thanks!

@onlyanegg

This comment has been minimized.

Show comment
Hide comment
@onlyanegg

onlyanegg Jul 19, 2017

Hi, i'd like to throw my name into the please-implement-this hat. Or maybe there's a different solution to my problem (docker noob here) that someone could point me to?

I'm trying to build an image based on the official centos/systemd image and provision it with Saltstack. This requires starting (and maybe restarting) the salt-minion daemon with systemd which can't be done (AFAIK) without privileged mode.

Hi, i'd like to throw my name into the please-implement-this hat. Or maybe there's a different solution to my problem (docker noob here) that someone could point me to?

I'm trying to build an image based on the official centos/systemd image and provision it with Saltstack. This requires starting (and maybe restarting) the salt-minion daemon with systemd which can't be done (AFAIK) without privileged mode.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 19, 2017

Member

@onlyanegg I think in that situation, Saltstack largely replaces the functionality of the builder; keep in mind that each RUN statement is executed in a new container, at which point the previous build container is stopped, and committed to an image / layer.

Have you considered performing the build by running a container, and committing the results (docker commit)?

Member

thaJeztah commented Jul 19, 2017

@onlyanegg I think in that situation, Saltstack largely replaces the functionality of the builder; keep in mind that each RUN statement is executed in a new container, at which point the previous build container is stopped, and committed to an image / layer.

Have you considered performing the build by running a container, and committing the results (docker commit)?

@onlyanegg

This comment has been minimized.

Show comment
Hide comment
@onlyanegg

onlyanegg Jul 19, 2017

Thanks for responding, @thaJeztah. I didn't realize that's what the RUN directive did. I did read most of this issue, so I am aware of the docker build -> docker run -> docker commit workaround, which is what I'll likely end up doing. I'm just more in favor of having a single file describe my image - seems neater. Maybe I can put all those steps in packer post-processors and then I'll have that.

Thanks for responding, @thaJeztah. I didn't realize that's what the RUN directive did. I did read most of this issue, so I am aware of the docker build -> docker run -> docker commit workaround, which is what I'll likely end up doing. I'm just more in favor of having a single file describe my image - seems neater. Maybe I can put all those steps in packer post-processors and then I'll have that.

@derberg

This comment has been minimized.

Show comment
Hide comment
@derberg

derberg Oct 9, 2017

Why is this one ignored so much? In times of containers, kubernetes and minikube, and usage of docker in CI and development environment unification, this functionality is really crucial.

derberg commented Oct 9, 2017

Why is this one ignored so much? In times of containers, kubernetes and minikube, and usage of docker in CI and development environment unification, this functionality is really crucial.

@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 9, 2017

Contributor

@onlyanegg you should be able to restart services without privileged mode. If you have a Dockerfile illustrating that (i.e., "the RUN command at line 8 of this Dockerfile doesn't work because it requires privileged mode") I would be more than happy to take a look!

@derberg precisely! In times of containers, CI, CD, it is important that build tools can be contained (in the security sense). If you allow privileged mode, you have to change dramatically how you use CI tools like Jenkins, Travis, Codeship, etc. Same question: if you have a Dockerfile that requires privileged mode, I would be happy to take a look to suggest alternatives.

Thank you!

Contributor

jpetazzo commented Oct 9, 2017

@onlyanegg you should be able to restart services without privileged mode. If you have a Dockerfile illustrating that (i.e., "the RUN command at line 8 of this Dockerfile doesn't work because it requires privileged mode") I would be more than happy to take a look!

@derberg precisely! In times of containers, CI, CD, it is important that build tools can be contained (in the security sense). If you allow privileged mode, you have to change dramatically how you use CI tools like Jenkins, Travis, Codeship, etc. Same question: if you have a Dockerfile that requires privileged mode, I would be happy to take a look to suggest alternatives.

Thank you!

@derberg

This comment has been minimized.

Show comment
Hide comment
@derberg

derberg Oct 9, 2017

@jpetazzo try to get a docker image with docker inside it:

FROM ubuntu:16.04

# Get dependencies for curl of the docker
RUN apt-get update && apt-get install -y \
    curl \
    sudo \
    bash \
    && rm -rf /var/lib/apt/lists/*

RUN curl -sSL https://get.docker.com/ | sh

Now build it and start it. After starting run service docker start to start the docker deamon. Then check the status of the service service docker status:

  • with privileged flag status is ok and you can start container without issues
  • without the flag, it never starts

derberg commented Oct 9, 2017

@jpetazzo try to get a docker image with docker inside it:

FROM ubuntu:16.04

# Get dependencies for curl of the docker
RUN apt-get update && apt-get install -y \
    curl \
    sudo \
    bash \
    && rm -rf /var/lib/apt/lists/*

RUN curl -sSL https://get.docker.com/ | sh

Now build it and start it. After starting run service docker start to start the docker deamon. Then check the status of the service service docker status:

  • with privileged flag status is ok and you can start container without issues
  • without the flag, it never starts
@derberg

This comment has been minimized.

Show comment
Hide comment
@derberg

derberg Oct 9, 2017

@jpetazzo ech, just noticed you are the creator of https://github.com/jpetazzo/dind :) so you are aware of docker in docker concept :)

Anyway, so you are aware that privileged flag is needed for run. So now you can imagine a group of people working on some environment and wanting to have a unified environment for development with already some preconfigured things inside, for example minikube with preinstalled components or anything else

derberg commented Oct 9, 2017

@jpetazzo ech, just noticed you are the creator of https://github.com/jpetazzo/dind :) so you are aware of docker in docker concept :)

Anyway, so you are aware that privileged flag is needed for run. So now you can imagine a group of people working on some environment and wanting to have a unified environment for development with already some preconfigured things inside, for example minikube with preinstalled components or anything else

@zerthimon

This comment has been minimized.

Show comment
Hide comment
@zerthimon

zerthimon Oct 9, 2017

So, is there a way to mount an NFS or SMB share in docker build yet ?

zerthimon commented Oct 9, 2017

So, is there a way to mount an NFS or SMB share in docker build yet ?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 9, 2017

Member

@derberg those steps won't work, even if the build container was running --privileged; the docker packages (and install script) are (for example) installing kernel packages on Ubuntu 16.04.
That's exactly the reason --privileged is a bad idea for docker build, because it would be able to make changes on the host.

Even though docker will need privileged when running, the installation itself does not need this; for example, here's the steps you'd run to install docker in your image;

docker build -t foo -<<'EOF'
FROM ubuntu:16.04

RUN apt-get update && apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common \
    && rm -rf /var/lib/apt/lists/*

RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable"
RUN apt-get update && apt-get install -y docker-ce \
    && rm -rf /var/lib/apt/lists/*
EOF

And you can run it just fine (I'm using --privileged here, but perhaps more fine-grained permissions are possible):

docker run -it --rm --privileged -v /var/lib/docker foo dockerd --debug
Member

thaJeztah commented Oct 9, 2017

@derberg those steps won't work, even if the build container was running --privileged; the docker packages (and install script) are (for example) installing kernel packages on Ubuntu 16.04.
That's exactly the reason --privileged is a bad idea for docker build, because it would be able to make changes on the host.

Even though docker will need privileged when running, the installation itself does not need this; for example, here's the steps you'd run to install docker in your image;

docker build -t foo -<<'EOF'
FROM ubuntu:16.04

RUN apt-get update && apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common \
    && rm -rf /var/lib/apt/lists/*

RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable"
RUN apt-get update && apt-get install -y docker-ce \
    && rm -rf /var/lib/apt/lists/*
EOF

And you can run it just fine (I'm using --privileged here, but perhaps more fine-grained permissions are possible):

docker run -it --rm --privileged -v /var/lib/docker foo dockerd --debug
@kraml

This comment has been minimized.

Show comment
Hide comment
@kraml

kraml Oct 10, 2017

Here is how I circumvent this issue, for people who really need to build a docker image in privileged mode. It may not solve all cases but can help in some.

This method need a Dockerfile, a docker-compse file, and a shell script.

The Dockerfile

It is the same as you would need to build the image. The only difference, stop at where you need do privileged operations, don't include them. They need to be run by the docker-compose, as a script. For example:

FROM ubuntu:16.04
RUN apt-get update && apt-get install <your packages>
# And more commands
......

## Below are the operations you intended to run in privileged mode when building the image, which does not work.
# More commands....
## But they now are moved to a separated shell script and it will be included in the image
COPY further-commands-to-run-in-privileged-mode.sh /

Those more commands that need to be run in privileged mode are now in further-commands-to-run-in-privileged-mode.sh. It is included in the image and later will be run by docker composer to finished the build process.

The docker compose file

The compose file is the key. It will build the image first from above Dockerfile, and start a container from that image in privileged mode, then you can do your privileged operation there. For example:

version: '3'

services:
  your_service:
    container_name: your_container
    # First build the image from the Dockerfile
    build:
      # Change this to where you keep above Dockerfile
      context: ../docker-build
    image: "your_image_name:your_image_tag"

    # Then start a container from the just built image in privileged mode to finish what's left
    entrypoint: /further-commands-to-run-in-privileged-mode.sh
    privileged: true

Build the image

Below steps can also be save in a shell script.

# First build the image and container(in privileged mode)
docker-compose -f docker-compose.yml up

# Then commit the temporary build container to a new image, change the ENTRYPOINT to what you want
docker commit \
    -c 'ENTRYPOINT ["/bin/bash"]' \
    <build container name> \
    <final image name>:<final image tag>

# Remove the temporary build container
docker rm <build container name>

kraml commented Oct 10, 2017

Here is how I circumvent this issue, for people who really need to build a docker image in privileged mode. It may not solve all cases but can help in some.

This method need a Dockerfile, a docker-compse file, and a shell script.

The Dockerfile

It is the same as you would need to build the image. The only difference, stop at where you need do privileged operations, don't include them. They need to be run by the docker-compose, as a script. For example:

FROM ubuntu:16.04
RUN apt-get update && apt-get install <your packages>
# And more commands
......

## Below are the operations you intended to run in privileged mode when building the image, which does not work.
# More commands....
## But they now are moved to a separated shell script and it will be included in the image
COPY further-commands-to-run-in-privileged-mode.sh /

Those more commands that need to be run in privileged mode are now in further-commands-to-run-in-privileged-mode.sh. It is included in the image and later will be run by docker composer to finished the build process.

The docker compose file

The compose file is the key. It will build the image first from above Dockerfile, and start a container from that image in privileged mode, then you can do your privileged operation there. For example:

version: '3'

services:
  your_service:
    container_name: your_container
    # First build the image from the Dockerfile
    build:
      # Change this to where you keep above Dockerfile
      context: ../docker-build
    image: "your_image_name:your_image_tag"

    # Then start a container from the just built image in privileged mode to finish what's left
    entrypoint: /further-commands-to-run-in-privileged-mode.sh
    privileged: true

Build the image

Below steps can also be save in a shell script.

# First build the image and container(in privileged mode)
docker-compose -f docker-compose.yml up

# Then commit the temporary build container to a new image, change the ENTRYPOINT to what you want
docker commit \
    -c 'ENTRYPOINT ["/bin/bash"]' \
    <build container name> \
    <final image name>:<final image tag>

# Remove the temporary build container
docker rm <build container name>
@derberg

This comment has been minimized.

Show comment
Hide comment
@derberg

derberg Oct 12, 2017

@thaJeztah I don't have a problem with installation, I have problem with starting the docker service during the build and pulling some images to have them available in the image out of the box.

Add the following script to your Dockerfile and you will see that docker service never gets up

#!/bin/bash

service docker start

sleep 20

service docker status

docker pull busybox

derberg commented Oct 12, 2017

@thaJeztah I don't have a problem with installation, I have problem with starting the docker service during the build and pulling some images to have them available in the image out of the box.

Add the following script to your Dockerfile and you will see that docker service never gets up

#!/bin/bash

service docker start

sleep 20

service docker status

docker pull busybox
@jpetazzo

This comment has been minimized.

Show comment
Hide comment
@jpetazzo

jpetazzo Oct 12, 2017

Contributor

@derberg OK, I see! Personally, if I wanted to include images in a dind container, I would download them (for instance with reg) and I would load them the first time that the container starts. Why? Because if you pull the images during the build, the image will work only if dind is started with the same storage driver. In other words, your image might or might not work on other machine.

Also–if your images are big (i.e. anything else than, say, busybox or alpine) you will end up with a really big DinD image...

I'd love to know more about your final use-case – because I'm sure we can help you to find a more efficient way than baking a huge DinD image :-)

(Otherwise, the solution proposed by @kraml is rather elegant!)

Contributor

jpetazzo commented Oct 12, 2017

@derberg OK, I see! Personally, if I wanted to include images in a dind container, I would download them (for instance with reg) and I would load them the first time that the container starts. Why? Because if you pull the images during the build, the image will work only if dind is started with the same storage driver. In other words, your image might or might not work on other machine.

Also–if your images are big (i.e. anything else than, say, busybox or alpine) you will end up with a really big DinD image...

I'd love to know more about your final use-case – because I'm sure we can help you to find a more efficient way than baking a huge DinD image :-)

(Otherwise, the solution proposed by @kraml is rather elegant!)

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
Contributor

cpuguy83 commented Oct 12, 2017

@derberg

This comment has been minimized.

Show comment
Hide comment
@derberg

derberg Oct 12, 2017

@jpetazzo we already run with such workaround build-run-commit, but yeah, it is still a workaround from my point of view. Use case is pretty specific related to kubernetes and minikube environment and there is nothing we can do for now. For now we were able to start minikube in docker only with docker as a daemon, using virtualbox or other vm drivers didn't work, so we are dependent on dind approach

derberg commented Oct 12, 2017

@jpetazzo we already run with such workaround build-run-commit, but yeah, it is still a workaround from my point of view. Use case is pretty specific related to kubernetes and minikube environment and there is nothing we can do for now. For now we were able to start minikube in docker only with docker as a daemon, using virtualbox or other vm drivers didn't work, so we are dependent on dind approach

@inthecloud247

This comment has been minimized.

Show comment
Hide comment
@inthecloud247

inthecloud247 Dec 5, 2017

Ran into this issue trying to build a image containing a legacy application (pretty normal use-case), where the installer tried to run sysctl commands and failed.

Coming back to this thread and reviewing the entire 4 years (!!!) of back and forth on the issue of how to add some kind of privileged capabilities to the docker build command, it appears that the available options in this situation are either a nasty bunch of sed commands to modify the installer to remove the sysctl calls, or a multi-stage build -> run -> commit pipeline. I agree with @derberg that 'build -> run -> commit' feels to be a workaround (imo a gross/hacky workaround) and I don't think that my use-case is that unique. Checking other threads I've seen plenty of people reporting issues with various application and database installations with failed docker build commands due to lack of privileges.

At this point, the docker run command supports extensive 'privileged' options, along with "fine grain control over the capabilities using --cap-add and --cap-drop". And so, I think objections on a security or technical basis are moot. If the privileged run option is added along with '--cap-add' and '--cap-drop', a security-conscious engineer could choose to limit a privileged build to only include the specific capabilities required for their build.

Ran into this issue trying to build a image containing a legacy application (pretty normal use-case), where the installer tried to run sysctl commands and failed.

Coming back to this thread and reviewing the entire 4 years (!!!) of back and forth on the issue of how to add some kind of privileged capabilities to the docker build command, it appears that the available options in this situation are either a nasty bunch of sed commands to modify the installer to remove the sysctl calls, or a multi-stage build -> run -> commit pipeline. I agree with @derberg that 'build -> run -> commit' feels to be a workaround (imo a gross/hacky workaround) and I don't think that my use-case is that unique. Checking other threads I've seen plenty of people reporting issues with various application and database installations with failed docker build commands due to lack of privileges.

At this point, the docker run command supports extensive 'privileged' options, along with "fine grain control over the capabilities using --cap-add and --cap-drop". And so, I think objections on a security or technical basis are moot. If the privileged run option is added along with '--cap-add' and '--cap-drop', a security-conscious engineer could choose to limit a privileged build to only include the specific capabilities required for their build.

@iahmad-khan

This comment has been minimized.

Show comment
Hide comment
@iahmad-khan

iahmad-khan Dec 5, 2017

Hi ,

I have already reported this before , same problem.

What about those who just want to run one container per VM with same user id on VM and container , using docker just as a packaging tool?

Is there still a security concern relating to this?

iahmad-khan commented Dec 5, 2017

Hi ,

I have already reported this before , same problem.

What about those who just want to run one container per VM with same user id on VM and container , using docker just as a packaging tool?

Is there still a security concern relating to this?

@krojew

This comment has been minimized.

Show comment
Hide comment
@krojew

krojew Jan 12, 2018

Ran into this problem. Could really use capabilities for build.

krojew commented Jan 12, 2018

Ran into this problem. Could really use capabilities for build.

@stanzgy

This comment has been minimized.

Show comment
Hide comment
@stanzgy

stanzgy Jan 12, 2018

Ran into this issue too.
It's very useful when using docker as CI/CD slaves which could require privileged permission on docker build to install the CI/CD build/test toolchains when building the slave docker image.
I'm waiting for this feature for years and really hope it could be supported in future.

stanzgy commented Jan 12, 2018

Ran into this issue too.
It's very useful when using docker as CI/CD slaves which could require privileged permission on docker build to install the CI/CD build/test toolchains when building the slave docker image.
I'm waiting for this feature for years and really hope it could be supported in future.

@maneamarius

This comment has been minimized.

Show comment
Hide comment
@maneamarius

maneamarius Jan 29, 2018

I really don't understand why there is so much pushback from devs regarding --privileged for docker image.
If the users want to shoot themselves in the foot, why not let them? Just put a warning message and that's it. There are already workarounds for achieving same thing, why not make it easier for the users that really need it??
It's been 4-5 years and has been no progress on this.
Just amazing...

maneamarius commented Jan 29, 2018

I really don't understand why there is so much pushback from devs regarding --privileged for docker image.
If the users want to shoot themselves in the foot, why not let them? Just put a warning message and that's it. There are already workarounds for achieving same thing, why not make it easier for the users that really need it??
It's been 4-5 years and has been no progress on this.
Just amazing...

@maneamarius

This comment has been minimized.

Show comment
Hide comment
@maneamarius

maneamarius Jan 29, 2018

As of today, not even this feature has been implemented yet:
RUN --cap-add=SYS_PTRACE
which would fit the needs of many users..

maneamarius commented Jan 29, 2018

As of today, not even this feature has been implemented yet:
RUN --cap-add=SYS_PTRACE
which would fit the needs of many users..

@maneamarius

This comment has been minimized.

Show comment
Hide comment
@maneamarius

maneamarius Jan 29, 2018

Could you please suggest how I can build this Dockerfile on Gentoo Linux host:

FROM gentoo/stage3-amd64
# Download and extract latest portage
RUN wget http://distfiles.gentoo.org/snapshots/portage-latest.tar.bz2 && \
    wget http://distfiles.gentoo.org/snapshots/portage-latest.tar.bz2.md5sum && \
    md5sum -c portage-latest.tar.bz2.md5sum 
RUN tar -xjvf portage-latest.tar.bz2 -C /usr
RUN emerge dev-lang/go

because I am getting this error when emerge-ing dev-lang/go:

##### Building Go bootstrap tool.
cmd/dist
 * /var/tmp/portage/sys-apps/sandbox-2.12/work/sandbox-2.12/libsandbox/trace.c:_do_ptrace():75: failure (Operation not permitted):
 * ISE:_do_ptrace: ptrace(PTRACE_TRACEME, ..., 0x0000000000000000, 0x0000000000000000): Operation not permitted
/usr/lib64/libsandbox.so(+0xb692)[0x7fd10e265692]
/usr/lib64/libsandbox.so(+0xb778)[0x7fd10e265778]
/usr/lib64/libsandbox.so(+0x6259)[0x7fd10e260259]
/usr/lib64/libsandbox.so(+0x6478)[0x7fd10e260478]
/usr/lib64/libsandbox.so(+0x7611)[0x7fd10e261611]
/usr/lib64/libsandbox.so(execve+0x3f)[0x7fd10e2634ff]
bash[0x41d8ff]
bash[0x41f387]
bash[0x420138]
bash[0x4219ce]
/proc/330/cmdline: bash ./make.bash 

 * ERROR: dev-lang/go-1.9.2::gentoo failed (compile phase):
 *   build failed
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 1034:  Called die
 * The specific snippet of code:
 *       ./make.bash || die "build failed"
 * 
 * If you need support, post the output of `emerge --info '=dev-lang/go-1.9.2::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-lang/go-1.9.2::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-lang/go-1.9.2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-lang/go-1.9.2/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-lang/go-1.9.2/work/go/src'
 * S: '/var/tmp/portage/dev-lang/go-1.9.2/work/go'

Could you please suggest how I can build this Dockerfile on Gentoo Linux host:

FROM gentoo/stage3-amd64
# Download and extract latest portage
RUN wget http://distfiles.gentoo.org/snapshots/portage-latest.tar.bz2 && \
    wget http://distfiles.gentoo.org/snapshots/portage-latest.tar.bz2.md5sum && \
    md5sum -c portage-latest.tar.bz2.md5sum 
RUN tar -xjvf portage-latest.tar.bz2 -C /usr
RUN emerge dev-lang/go

because I am getting this error when emerge-ing dev-lang/go:

##### Building Go bootstrap tool.
cmd/dist
 * /var/tmp/portage/sys-apps/sandbox-2.12/work/sandbox-2.12/libsandbox/trace.c:_do_ptrace():75: failure (Operation not permitted):
 * ISE:_do_ptrace: ptrace(PTRACE_TRACEME, ..., 0x0000000000000000, 0x0000000000000000): Operation not permitted
/usr/lib64/libsandbox.so(+0xb692)[0x7fd10e265692]
/usr/lib64/libsandbox.so(+0xb778)[0x7fd10e265778]
/usr/lib64/libsandbox.so(+0x6259)[0x7fd10e260259]
/usr/lib64/libsandbox.so(+0x6478)[0x7fd10e260478]
/usr/lib64/libsandbox.so(+0x7611)[0x7fd10e261611]
/usr/lib64/libsandbox.so(execve+0x3f)[0x7fd10e2634ff]
bash[0x41d8ff]
bash[0x41f387]
bash[0x420138]
bash[0x4219ce]
/proc/330/cmdline: bash ./make.bash 

 * ERROR: dev-lang/go-1.9.2::gentoo failed (compile phase):
 *   build failed
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 1034:  Called die
 * The specific snippet of code:
 *       ./make.bash || die "build failed"
 * 
 * If you need support, post the output of `emerge --info '=dev-lang/go-1.9.2::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-lang/go-1.9.2::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-lang/go-1.9.2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-lang/go-1.9.2/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-lang/go-1.9.2/work/go/src'
 * S: '/var/tmp/portage/dev-lang/go-1.9.2/work/go'

pm98zz-c added a commit to codeenigma/ce-vm-base that referenced this issue Feb 18, 2018

@acouch acouch referenced this issue Apr 24, 2018

Merged

Create Circle Base #2

@AMD-NICK

This comment has been minimized.

Show comment
Hide comment
@AMD-NICK

AMD-NICK Jul 20, 2018

How can I run it without --cap-add=SYS_ADMIN --device /dev/fuse or --privileged?

RUN apt-get -y install unionfs-fuse
RUN unionfs-fuse -o cow dir1=RW:dir2=RO dir3/

I can do it with a separate bash file in entrypoint, but I need single Dockerfile

How can I run it without --cap-add=SYS_ADMIN --device /dev/fuse or --privileged?

RUN apt-get -y install unionfs-fuse
RUN unionfs-fuse -o cow dir1=RW:dir2=RO dir3/

I can do it with a separate bash file in entrypoint, but I need single Dockerfile

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 20, 2018

Member

@AMD-NICK what's your expectation of the RUN unionfs-fuse ... line during build? Even if that worked, it would only have the filesystem mounted during that single RUN, and be gone in the next step.

Member

thaJeztah commented Jul 20, 2018

@AMD-NICK what's your expectation of the RUN unionfs-fuse ... line during build? Even if that worked, it would only have the filesystem mounted during that single RUN, and be gone in the next step.

@AMD-NICK

This comment has been minimized.

Show comment
Hide comment
@AMD-NICK

AMD-NICK Jul 20, 2018

@thaJeztah it's hard to explain for me. I am trying to modify this repo. Can I just skip this line on the building?

@thaJeztah it's hard to explain for me. I am trying to modify this repo. Can I just skip this line on the building?

@dn2k-dev

This comment has been minimized.

Show comment
Hide comment
@dn2k-dev

dn2k-dev Jul 23, 2018

Hi

Randomly the docker build choose an hostname starting zero '0' which breaks our application, I tried to run "hostname" in such case inside my DockerFile but faced the same issue.

I also would like to have an option to run the docker build with RUNP or get an option to choose the hostname during build.

Hi

Randomly the docker build choose an hostname starting zero '0' which breaks our application, I tried to run "hostname" in such case inside my DockerFile but faced the same issue.

I also would like to have an option to run the docker build with RUNP or get an option to choose the hostname during build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment